ConcurrentRebalancersTest.TwoConcurrentRebalancers/1: /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 08:18:48.323661  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14124
I20240502 08:18:48.333635 14246 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:18:48.334493 14270 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.336072 14401 connection.cc:535] client connection to 127.6.151.70:45703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.336300 14534 connection.cc:535] client connection to 127.6.151.70:45703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.337803 19951 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:48.336748 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.151.70:45703 (local address 127.6.151.67:48349)
0502 08:18:48.336960 (+   212us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:48.337035 (+    75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":123}
W20240502 08:18:48.338927 14401 consensus_peers.cc:485] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.339279 14534 consensus_peers.cc:485] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.339435 14139 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.340482 14271 consensus_peers.cc:462] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111)
W20240502 08:18:48.354748 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.371646 14271 consensus_peers.cc:462] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111)
W20240502 08:18:48.425154 14270 consensus_peers.cc:485] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.428287 14271 consensus_peers.cc:485] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.445668 14534 consensus_peers.cc:485] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.458776 14270 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.481081 14270 consensus_peers.cc:462] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.493229 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:48.493268 14271 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.517007 14533 consensus_peers.cc:485] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.530458 14534 consensus_peers.cc:485] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.555467 14401 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.570508 14049 catalog_manager.cc:4573] TS 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6272b2f3a9504e01832cb31f5b322980: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.591276 16336 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.618366 14400 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.628896 14140 connection.cc:535] client connection to 127.6.151.70:45703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.632824 14140 consensus_peers.cc:485] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.633023 14140 consensus_peers.cc:485] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.634307 14140 consensus_peers.cc:485] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.634485 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.634608 14140 consensus_peers.cc:485] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.640947 14140 consensus_peers.cc:485] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:48.643085  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14124
W20240502 08:18:48.682850 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.688782 14534 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.701601 14271 consensus_peers.cc:485] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.706326 14402 connection.cc:535] server connection from 127.6.151.65:34957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.706388 14400 connection.cc:535] client connection to 127.6.151.65:43987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.706480 16335 connection.cc:535] server connection from 127.6.151.65:49557 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.706951 14270 connection.cc:535] server connection from 127.6.151.65:60975 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:48.707340  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14255
W20240502 08:18:48.708068 14400 consensus_peers.cc:485] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Couldn't send request to peer b19bedd6f7f94bed9c727f6c7210cc8a. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:43987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.708184 14400 consensus_peers.cc:485] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Couldn't send request to peer b19bedd6f7f94bed9c727f6c7210cc8a. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:43987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:48.713766 14376 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:18:48.879817 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:48.983848  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14255
W20240502 08:18:49.023502 14401 connection.cc:535] server connection from 127.6.151.66:41887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:49.023854 14050 connection.cc:535] server connection from 127.6.151.66:47089 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:49.024077 14535 connection.cc:535] server connection from 127.6.151.66:51175 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:49.024124  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14385
I20240502 08:18:49.030411 14506 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:18:49.101513 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:49.206802 14535 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): Couldn't send request to peer 8ae1cf63e5bd46cf9a4ba89651971dde. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:44269: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:49.219919 14535 consensus_peers.cc:485] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): Couldn't send request to peer 8ae1cf63e5bd46cf9a4ba89651971dde. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:44269: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:49.267400 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:49.301919 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:49.305784  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14385
W20240502 08:18:49.348984 14535 connection.cc:535] client connection to 127.6.151.67:39913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:49.349702  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14517
W20240502 08:18:49.350342 14535 consensus_peers.cc:485] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b -> Peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): Couldn't send request to peer 32e074cd0dfd4311adc4e2a0f5d1ad8a. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.67:39913: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:49.355998 14639 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:49.401188 20088 raft_consensus.cc:482] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5b84b46faddb47fcae86cc5f3f478bf1)
I20240502 08:18:49.401403 20088 raft_consensus.cc:504] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:49.402483 20088 leader_election.cc:288] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703)
W20240502 08:18:49.404776 16337 leader_election.cc:334] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): Network error: Client connection negotiation failed: client connection to 127.6.151.67:39913: connect: Connection refused (error 111)
W20240502 08:18:49.406193 16336 leader_election.cc:334] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111)
I20240502 08:18:49.406399 16336 leader_election.cc:302] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b3dfd9d6472342689afd86203f7afe5f; no voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:49.406661 16335 leader_election.cc:334] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Network error: Client connection negotiation failed: client connection to 127.6.151.65:43987: connect: Connection refused (error 111)
I20240502 08:18:49.406739 20088 raft_consensus.cc:2726] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:49.618683  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14517
W20240502 08:18:49.662395 16335 connection.cc:535] server connection from 127.6.151.68:50817 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:49.663074  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 16318
W20240502 08:18:49.663550 16337 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f -> Peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): Couldn't send request to peer e79a5862de19490b89a9ea97860d776b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.68:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:49.668690 16440 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:49.865217  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 16318
I20240502 08:18:49.899910  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14034
I20240502 08:18:49.901093 14092 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:50.151006  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14034
2024-05-02T08:18:50Z chronyd exiting
I20240502 08:18:50.214078  6749 test_util.cc:175] -----------------------------------------------
I20240502 08:18:50.214185  6749 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0

Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN      ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 08:16:30.539790  6749 test_util.cc:268] Using random seed: 946630621
I20240502 08:16:30.551070  6749 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:16:30.551153  6749 ts_itest-base.cc:113] --------------
I20240502 08:16:30.551198  6749 ts_itest-base.cc:114] 6 tablet servers
I20240502 08:16:30.551239  6749 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:16:30.551282  6749 ts_itest-base.cc:116] --------------
2024-05-02T08:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:16:30Z Disabled control of system clock
I20240502 08:16:30.576556  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:34373
--webserver_interface=127.6.151.126
--webserver_port=0
--builtin_ntp_servers=127.6.151.84:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.151.126:34373
--location_mapping_cmd=/tmp/dist-test-taskOUdoRy/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 08:16:30.891713  6761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:30.892017  6761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:30.892096  6761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:30.899453  6761 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:30.899578  6761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:16:30.899624  6761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:30.899679  6761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:16:30.899729  6761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:16:30.908710  6761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskOUdoRy/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.6.151.126:34373
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.151.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:30.910017  6761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:30.911700  6761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:30.923123  6767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:30.924141  6766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:30.925015  6769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:30.925843  6761 server_base.cc:1034] running on GCE node
I20240502 08:16:30.926604  6761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:30.928077  6761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:30.929288  6761 hybrid_clock.cc:648] HybridClock initialized: now 1714637790929250 us; error 32 us; skew 500 ppm
I20240502 08:16:30.929618  6761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:30.937678  6761 webserver.cc:460] Webserver started at http://127.6.151.126:36717/ using document root <none> and password file <none>
I20240502 08:16:30.938501  6761 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:30.938630  6761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:30.938987  6761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:30.941653  6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a10cf1e9c4d74698a44f15d67b76e040"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-x3qn"
I20240502 08:16:30.942420  6761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a10cf1e9c4d74698a44f15d67b76e040"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-x3qn"
I20240502 08:16:30.947839  6761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.001s
I20240502 08:16:30.951207  6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:30.952608  6761 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:16:30.952797  6761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a10cf1e9c4d74698a44f15d67b76e040"
format_stamp: "Formatted at 2024-05-02 08:16:30 on dist-test-slave-x3qn"
I20240502 08:16:30.953004  6761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:30.987926  6761 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:16:30.988796  6761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:30.989070  6761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:31.006758  6761 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.126:34373
I20240502 08:16:31.006779  6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.126:34373 every 8 connection(s)
I20240502 08:16:31.008131  6761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:16:31.011801  6827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:31.013232  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 6761
I20240502 08:16:31.013481  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:16:31.021981  6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040: Bootstrap starting.
I20240502 08:16:31.024858  6827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:31.025826  6827 log.cc:824] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:31.028461  6827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040: No bootstrap required, opened a new log
I20240502 08:16:31.035084  6827 raft_consensus.cc:348] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10cf1e9c4d74698a44f15d67b76e040" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 34373 } }
I20240502 08:16:31.035434  6827 raft_consensus.cc:374] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:31.035563  6827 raft_consensus.cc:729] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a10cf1e9c4d74698a44f15d67b76e040, State: Initialized, Role: FOLLOWER
I20240502 08:16:31.036156  6827 consensus_queue.cc:260] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10cf1e9c4d74698a44f15d67b76e040" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 34373 } }
I20240502 08:16:31.036363  6827 raft_consensus.cc:388] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:16:31.036507  6827 raft_consensus.cc:482] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:16:31.036666  6827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:31.039462  6827 raft_consensus.cc:504] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10cf1e9c4d74698a44f15d67b76e040" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 34373 } }
I20240502 08:16:31.040022  6827 leader_election.cc:302] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a10cf1e9c4d74698a44f15d67b76e040; no voters: 
I20240502 08:16:31.040679  6827 leader_election.cc:288] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:16:31.040810  6832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:31.043228  6827 sys_catalog.cc:564] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:16:31.043661  6832 raft_consensus.cc:686] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [term 1 LEADER]: Becoming Leader. State: Replica: a10cf1e9c4d74698a44f15d67b76e040, State: Running, Role: LEADER
I20240502 08:16:31.044219  6832 consensus_queue.cc:237] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10cf1e9c4d74698a44f15d67b76e040" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 34373 } }
I20240502 08:16:31.047333  6834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a10cf1e9c4d74698a44f15d67b76e040. Latest consensus state: current_term: 1 leader_uuid: "a10cf1e9c4d74698a44f15d67b76e040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10cf1e9c4d74698a44f15d67b76e040" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 34373 } } }
I20240502 08:16:31.047803  6834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:31.048548  6833 sys_catalog.cc:455] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a10cf1e9c4d74698a44f15d67b76e040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a10cf1e9c4d74698a44f15d67b76e040" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 34373 } } }
I20240502 08:16:31.048868  6833 sys_catalog.cc:458] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040 [sys.catalog]: This master's current role is: LEADER
I20240502 08:16:31.049599  6841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:16:31.055054  6841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:16:31.063074  6841 catalog_manager.cc:1347] Generated new cluster ID: 76d42fea4e8b4484b7665f8448195f36
I20240502 08:16:31.063207  6841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:16:31.081085  6841 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:16:31.082216  6841 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:16:31.090600  6841 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a10cf1e9c4d74698a44f15d67b76e040: Generated new TSK 0
I20240502 08:16:31.091454  6841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:16:31.099721  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.65:0
--local_ip_for_outbound_sockets=127.6.151.65
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--builtin_ntp_servers=127.6.151.84:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:16:31.432034  6851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:31.432328  6851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:31.432452  6851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:31.440073  6851 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:31.440202  6851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:31.440369  6851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.65
I20240502 08:16:31.449786  6851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.65
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:31.451236  6851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:31.452950  6851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:31.465694  6857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:31.466428  6859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:31.467455  6856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:31.468494  6851 server_base.cc:1034] running on GCE node
I20240502 08:16:31.469107  6851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:31.470093  6851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:31.471314  6851 hybrid_clock.cc:648] HybridClock initialized: now 1714637791471262 us; error 33 us; skew 500 ppm
I20240502 08:16:31.471647  6851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:31.473935  6851 webserver.cc:460] Webserver started at http://127.6.151.65:36555/ using document root <none> and password file <none>
I20240502 08:16:31.474745  6851 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:31.474871  6851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:31.475203  6851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:31.477797  6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cc93fe73601544fe8a8aa013f8de095b"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-x3qn"
I20240502 08:16:31.478562  6851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cc93fe73601544fe8a8aa013f8de095b"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-x3qn"
I20240502 08:16:31.483737  6851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:16:31.487210  6865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:31.488596  6851 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 08:16:31.488759  6851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cc93fe73601544fe8a8aa013f8de095b"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-x3qn"
I20240502 08:16:31.488965  6851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:31.515784  6851 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:16:31.516673  6851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:31.516983  6851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:31.517983  6851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:31.519613  6851 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:31.519711  6851 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:31.519829  6851 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:31.519901  6851 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:31.552626  6851 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.65:40359
I20240502 08:16:31.552695  6977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.65:40359 every 8 connection(s)
I20240502 08:16:31.554034  6851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:16:31.556897  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 6851
I20240502 08:16:31.557152  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:16:31.560233  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.66:0
--local_ip_for_outbound_sockets=127.6.151.66
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--builtin_ntp_servers=127.6.151.84:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:31.567107  6978 heartbeater.cc:347] Connected to a master server at 127.6.151.126:34373
I20240502 08:16:31.567399  6978 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:31.568222  6978 heartbeater.cc:511] Master 127.6.151.126:34373 requested a full tablet report, sending...
I20240502 08:16:31.612275  6792 ts_manager.cc:194] Registered new tserver with Master: cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:31.614022  6792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.65:43843
W20240502 08:16:31.886180  6982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:31.886507  6982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:31.886679  6982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:31.894393  6982 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:31.894549  6982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:31.894742  6982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.66
I20240502 08:16:31.904140  6982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.66
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:31.905506  6982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:31.907263  6982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:31.920264  6988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:31.921164  6989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:31.922081  6991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:31.922863  6982 server_base.cc:1034] running on GCE node
I20240502 08:16:31.923493  6982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:31.924454  6982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:31.925663  6982 hybrid_clock.cc:648] HybridClock initialized: now 1714637791925606 us; error 45 us; skew 500 ppm
I20240502 08:16:31.926012  6982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:31.928364  6982 webserver.cc:460] Webserver started at http://127.6.151.66:34451/ using document root <none> and password file <none>
I20240502 08:16:31.929081  6982 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:31.929175  6982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:31.929458  6982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:31.932047  6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b8432324fa494be3a41163adc25b0865"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-x3qn"
I20240502 08:16:31.932760  6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b8432324fa494be3a41163adc25b0865"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-x3qn"
I20240502 08:16:31.937906  6982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 08:16:31.941431  6997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:31.942848  6982 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:16:31.943032  6982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b8432324fa494be3a41163adc25b0865"
format_stamp: "Formatted at 2024-05-02 08:16:31 on dist-test-slave-x3qn"
I20240502 08:16:31.943231  6982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:31.965106  6982 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:16:31.965961  6982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:31.966248  6982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:31.967278  6982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:31.968780  6982 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:31.968878  6982 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:31.969003  6982 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:31.969075  6982 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:32.000411  6982 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.66:46195
I20240502 08:16:32.000463  7109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.66:46195 every 8 connection(s)
I20240502 08:16:32.001688  6982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:16:32.007769  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 6982
I20240502 08:16:32.007992  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:16:32.011468  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:0
--local_ip_for_outbound_sockets=127.6.151.67
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--builtin_ntp_servers=127.6.151.84:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:32.015709  7110 heartbeater.cc:347] Connected to a master server at 127.6.151.126:34373
I20240502 08:16:32.016050  7110 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.017061  7110 heartbeater.cc:511] Master 127.6.151.126:34373 requested a full tablet report, sending...
I20240502 08:16:32.059815  6792 ts_manager.cc:194] Registered new tserver with Master: b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:32.060742  6792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.66:41679
W20240502 08:16:32.348067  7113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:32.348384  7113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:32.348516  7113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:32.356123  7113 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:32.356252  7113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:32.356411  7113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:16:32.365680  7113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:32.367053  7113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:32.368752  7113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:32.381337  7120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.382042  7119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.382769  7122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:32.383962  7113 server_base.cc:1034] running on GCE node
I20240502 08:16:32.384572  7113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:32.385511  7113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:32.386751  7113 hybrid_clock.cc:648] HybridClock initialized: now 1714637792386694 us; error 48 us; skew 500 ppm
I20240502 08:16:32.387063  7113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:32.389281  7113 webserver.cc:460] Webserver started at http://127.6.151.67:38479/ using document root <none> and password file <none>
I20240502 08:16:32.390000  7113 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:32.390120  7113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:32.390425  7113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:32.393049  7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "136680942cc947b0b19be2f50bb114eb"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-x3qn"
I20240502 08:16:32.393836  7113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "136680942cc947b0b19be2f50bb114eb"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-x3qn"
I20240502 08:16:32.399317  7113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20240502 08:16:32.404192  7128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:32.405982  7113 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 08:16:32.406157  7113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "136680942cc947b0b19be2f50bb114eb"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-x3qn"
I20240502 08:16:32.406425  7113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:32.418689  7113 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:16:32.419661  7113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:32.419987  7113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:32.421180  7113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:32.423087  7113 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:32.423184  7113 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:32.423266  7113 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:32.423324  7113 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:32.454022  7113 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:36791
I20240502 08:16:32.454078  7240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:36791 every 8 connection(s)
I20240502 08:16:32.455633  7113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:16:32.462471  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 7113
I20240502 08:16:32.462714  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:16:32.465967  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:0
--local_ip_for_outbound_sockets=127.6.151.68
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--builtin_ntp_servers=127.6.151.84:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:32.469591  7241 heartbeater.cc:347] Connected to a master server at 127.6.151.126:34373
I20240502 08:16:32.470028  7241 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.471175  7241 heartbeater.cc:511] Master 127.6.151.126:34373 requested a full tablet report, sending...
I20240502 08:16:32.532338  6792 ts_manager.cc:194] Registered new tserver with Master: 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:32.533620  6792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:47669
I20240502 08:16:32.617168  6978 heartbeater.cc:503] Master 127.6.151.126:34373 was elected leader, sending a full tablet report...
W20240502 08:16:32.805939  7244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:32.806262  7244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:32.806388  7244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:32.813802  7244 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:32.813925  7244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:32.814054  7244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:16:32.822986  7244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:32.824241  7244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:32.825879  7244 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:32.838349  7253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.839188  7251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:32.839816  7250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:32.840934  7244 server_base.cc:1034] running on GCE node
I20240502 08:16:32.841557  7244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:32.842515  7244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:32.843724  7244 hybrid_clock.cc:648] HybridClock initialized: now 1714637792843670 us; error 51 us; skew 500 ppm
I20240502 08:16:32.844031  7244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:32.846335  7244 webserver.cc:460] Webserver started at http://127.6.151.68:40161/ using document root <none> and password file <none>
I20240502 08:16:32.847126  7244 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:32.847230  7244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:32.847568  7244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:32.850100  7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-x3qn"
I20240502 08:16:32.850914  7244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-x3qn"
I20240502 08:16:32.855793  7244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:16:32.859117  7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:32.860420  7244 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 08:16:32.860565  7244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
format_stamp: "Formatted at 2024-05-02 08:16:32 on dist-test-slave-x3qn"
I20240502 08:16:32.860757  7244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:32.880506  7244 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:16:32.881350  7244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:32.881623  7244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:32.882615  7244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:32.884145  7244 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:32.884229  7244 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:32.884341  7244 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:32.884407  7244 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:32.917147  7244 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:34071
I20240502 08:16:32.917228  7371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:34071 every 8 connection(s)
I20240502 08:16:32.918491  7244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:16:32.924088  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 7244
I20240502 08:16:32.924270  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:16:32.927855  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:0
--local_ip_for_outbound_sockets=127.6.151.69
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--builtin_ntp_servers=127.6.151.84:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:32.932687  7372 heartbeater.cc:347] Connected to a master server at 127.6.151.126:34373
I20240502 08:16:32.932987  7372 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:32.933842  7372 heartbeater.cc:511] Master 127.6.151.126:34373 requested a full tablet report, sending...
I20240502 08:16:32.979615  6792 ts_manager.cc:194] Registered new tserver with Master: 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:32.980841  6792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:40407
I20240502 08:16:33.064167  7110 heartbeater.cc:503] Master 127.6.151.126:34373 was elected leader, sending a full tablet report...
W20240502 08:16:33.262825  7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:33.263132  7375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:33.263278  7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:33.270951  7375 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:33.271106  7375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:33.271279  7375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:16:33.280447  7375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:33.281817  7375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:33.283608  7375 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:33.296257  7381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.297170  7382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.298024  7384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:33.298959  7375 server_base.cc:1034] running on GCE node
I20240502 08:16:33.299618  7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:33.300612  7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:33.301846  7375 hybrid_clock.cc:648] HybridClock initialized: now 1714637793301799 us; error 37 us; skew 500 ppm
I20240502 08:16:33.302186  7375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:33.304569  7375 webserver.cc:460] Webserver started at http://127.6.151.69:33173/ using document root <none> and password file <none>
I20240502 08:16:33.305326  7375 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:33.305449  7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:33.305773  7375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:33.308362  7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a5b21a0f245140b293e607e8f554fa1f"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-x3qn"
I20240502 08:16:33.309109  7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a5b21a0f245140b293e607e8f554fa1f"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-x3qn"
I20240502 08:16:33.314314  7375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.005s
I20240502 08:16:33.317740  7390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:33.319126  7375 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:16:33.319279  7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a5b21a0f245140b293e607e8f554fa1f"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-x3qn"
I20240502 08:16:33.319470  7375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:33.334117  7375 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:16:33.334965  7375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:33.335227  7375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:33.336115  7375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:33.337493  7375 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:33.337589  7375 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:33.337702  7375 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:33.337774  7375 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:33.369803  7375 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:44083
I20240502 08:16:33.369853  7502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:44083 every 8 connection(s)
I20240502 08:16:33.371184  7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:16:33.376972  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 7375
I20240502 08:16:33.377189  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:16:33.380632  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.70:0
--local_ip_for_outbound_sockets=127.6.151.70
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--builtin_ntp_servers=127.6.151.84:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:16:33.384399  7503 heartbeater.cc:347] Connected to a master server at 127.6.151.126:34373
I20240502 08:16:33.384691  7503 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:33.385520  7503 heartbeater.cc:511] Master 127.6.151.126:34373 requested a full tablet report, sending...
I20240502 08:16:33.429982  6792 ts_manager.cc:194] Registered new tserver with Master: a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083)
I20240502 08:16:33.430958  6792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:41409
I20240502 08:16:33.536796  7241 heartbeater.cc:503] Master 127.6.151.126:34373 was elected leader, sending a full tablet report...
W20240502 08:16:33.712642  7506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:33.712931  7506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:16:33.713017  7506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:33.720474  7506 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:16:33.720605  7506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:16:33.720741  7506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.70
I20240502 08:16:33.729862  7506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:34373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.70
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:16:33.731192  7506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:16:33.732861  7506 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:16:33.745682  7513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.746572  7512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:16:33.747378  7515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:16:33.748275  7506 server_base.cc:1034] running on GCE node
I20240502 08:16:33.748885  7506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:16:33.749902  7506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:16:33.751097  7506 hybrid_clock.cc:648] HybridClock initialized: now 1714637793751049 us; error 43 us; skew 500 ppm
I20240502 08:16:33.751401  7506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:16:33.753667  7506 webserver.cc:460] Webserver started at http://127.6.151.70:44779/ using document root <none> and password file <none>
I20240502 08:16:33.754453  7506 fs_manager.cc:359] Metadata directory not provided
I20240502 08:16:33.754572  7506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:16:33.754905  7506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:16:33.757555  7506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "881e59a197394322ade814528d5e2cdf"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-x3qn"
I20240502 08:16:33.758368  7506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "881e59a197394322ade814528d5e2cdf"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-x3qn"
I20240502 08:16:33.763478  7506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:16:33.766984  7521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:33.768380  7506 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 08:16:33.768581  7506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "881e59a197394322ade814528d5e2cdf"
format_stamp: "Formatted at 2024-05-02 08:16:33 on dist-test-slave-x3qn"
I20240502 08:16:33.768774  7506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:16:33.795696  7506 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:16:33.796561  7506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:16:33.796855  7506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:16:33.797842  7506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:16:33.799312  7506 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:16:33.799407  7506 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:33.799532  7506 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:16:33.799607  7506 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:16:33.831354  7506 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.70:36909
I20240502 08:16:33.831420  7633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.70:36909 every 8 connection(s)
I20240502 08:16:33.832649  7506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:16:33.840317  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 7506
I20240502 08:16:33.840519  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:16:33.845666  7634 heartbeater.cc:347] Connected to a master server at 127.6.151.126:34373
I20240502 08:16:33.845945  7634 heartbeater.cc:464] Registering TS with master...
I20240502 08:16:33.846836  7634 heartbeater.cc:511] Master 127.6.151.126:34373 requested a full tablet report, sending...
I20240502 08:16:33.889724  6792 ts_manager.cc:194] Registered new tserver with Master: 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:33.890609  6792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.70:55789
I20240502 08:16:33.896008  6749 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 08:16:33.916539  6749 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 08:16:33.983649  7372 heartbeater.cc:503] Master 127.6.151.126:34373 was elected leader, sending a full tablet report...
I20240502 08:16:34.433749  7503 heartbeater.cc:503] Master 127.6.151.126:34373 was elected leader, sending a full tablet report...
I20240502 08:16:34.893409  7634 heartbeater.cc:503] Master 127.6.151.126:34373 was elected leader, sending a full tablet report...
I20240502 08:16:37.670626  6792 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:37.723361  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 44c06a7143db48fd98fa731b0baa161a (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.724637  7176 tablet_service.cc:1467] Processing CreateTablet for tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.725268  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c06a7143db48fd98fa731b0baa161a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.725739  7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.729045  7569 tablet_service.cc:1467] Processing CreateTablet for tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.730571  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.733422  6913 tablet_service.cc:1467] Processing CreateTablet for tablet c226114065e6429b8fef55539c315388 (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.734966  6912 tablet_service.cc:1467] Processing CreateTablet for tablet 44c06a7143db48fd98fa731b0baa161a (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.736399  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c226114065e6429b8fef55539c315388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.737025  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c06a7143db48fd98fa731b0baa161a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.739653  7307 tablet_service.cc:1467] Processing CreateTablet for tablet c226114065e6429b8fef55539c315388 (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.741226  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c226114065e6429b8fef55539c315388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.741189  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 44c06a7143db48fd98fa731b0baa161a (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.742996  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44c06a7143db48fd98fa731b0baa161a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.742080  7438 tablet_service.cc:1467] Processing CreateTablet for tablet c226114065e6429b8fef55539c315388 (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.743875  7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c226114065e6429b8fef55539c315388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.768628  7660 tablet_bootstrap.cc:492] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:37.778225  7658 tablet_bootstrap.cc:492] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:37.778928  7660 tablet_bootstrap.cc:654] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.779810  7659 tablet_bootstrap.cc:492] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:37.780457  7660 log.cc:824] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.782194  7045 tablet_service.cc:1467] Processing CreateTablet for tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=130cf18b8d224a6eabf796831bec33e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:37.783403  7659 tablet_bootstrap.cc:654] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.783435  7660 tablet_bootstrap.cc:492] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:37.783725  7660 ts_tablet_manager.cc:1367] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.015s	user 0.007s	sys 0.000s
I20240502 08:16:37.783867  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:37.784505  7658 tablet_bootstrap.cc:654] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.784696  7659 log.cc:824] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.785701  7658 log.cc:824] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.793953  7659 tablet_bootstrap.cc:492] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:37.794329  7659 ts_tablet_manager.cc:1367] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.003s
I20240502 08:16:37.795501  7661 tablet_bootstrap.cc:492] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:37.795547  7660 raft_consensus.cc:348] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.796308  7660 raft_consensus.cc:374] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.796496  7660 raft_consensus.cc:729] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.797618  7660 consensus_queue.cc:260] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.804469  7660 ts_tablet_manager.cc:1398] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.021s	user 0.014s	sys 0.005s
I20240502 08:16:37.804970  7660 tablet_bootstrap.cc:492] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:37.804765  7659 raft_consensus.cc:348] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.805271  7659 raft_consensus.cc:374] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.805310  7658 tablet_bootstrap.cc:492] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:37.805456  7659 raft_consensus.cc:729] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.805603  7658 ts_tablet_manager.cc:1367] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.008s
I20240502 08:16:37.806358  7659 consensus_queue.cc:260] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.806797  7660 tablet_bootstrap.cc:654] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.812127  7662 tablet_bootstrap.cc:492] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:37.814471  7658 raft_consensus.cc:348] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:37.814942  7659 ts_tablet_manager.cc:1398] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.020s	user 0.017s	sys 0.001s
I20240502 08:16:37.815027  7658 raft_consensus.cc:374] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.815127  7660 tablet_bootstrap.cc:492] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:37.815174  7658 raft_consensus.cc:729] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.815318  7660 ts_tablet_manager.cc:1367] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.009s
I20240502 08:16:37.815867  7662 tablet_bootstrap.cc:654] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.816053  7658 consensus_queue.cc:260] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:37.816339  7660 raft_consensus.cc:348] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.816596  7660 raft_consensus.cc:374] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.816735  7660 raft_consensus.cc:729] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.817018  7660 consensus_queue.cc:260] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.817744  7660 ts_tablet_manager.cc:1398] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:16:37.819836  7661 tablet_bootstrap.cc:654] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.821439  7658 ts_tablet_manager.cc:1398] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.016s	user 0.016s	sys 0.000s
I20240502 08:16:37.823060  7661 log.cc:824] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.823319  7662 log.cc:824] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb: Log is configured to *not* fsync() on all Append() calls
I20240502 08:16:37.827359  7672 tablet_bootstrap.cc:492] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:37.830435  7662 tablet_bootstrap.cc:492] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:37.830758  7662 ts_tablet_manager.cc:1367] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.000s
I20240502 08:16:37.830807  7661 tablet_bootstrap.cc:492] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:37.830896  7672 tablet_bootstrap.cc:654] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.831049  7661 ts_tablet_manager.cc:1367] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.036s	user 0.006s	sys 0.005s
I20240502 08:16:37.835876  7672 log.cc:824] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865: Log is configured to *not* fsync() on all Append() calls
W20240502 08:16:37.836999  7635 tablet.cc:2369] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:37.839814  7661 raft_consensus.cc:348] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.840384  7661 raft_consensus.cc:374] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.840539  7661 raft_consensus.cc:729] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.840703  7662 raft_consensus.cc:348] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:37.841151  7662 raft_consensus.cc:374] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.841285  7662 raft_consensus.cc:729] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.841955  7662 consensus_queue.cc:260] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:37.841535  7661 consensus_queue.cc:260] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.845134  7661 ts_tablet_manager.cc:1398] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.014s	user 0.012s	sys 0.001s
I20240502 08:16:37.845589  7661 tablet_bootstrap.cc:492] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:37.845969  7672 tablet_bootstrap.cc:492] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:37.847024  7662 ts_tablet_manager.cc:1398] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.016s	user 0.013s	sys 0.000s
I20240502 08:16:37.847400  7661 tablet_bootstrap.cc:654] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.847790  7672 ts_tablet_manager.cc:1367] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.007s
I20240502 08:16:37.848567  7662 tablet_bootstrap.cc:492] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:37.850533  7662 tablet_bootstrap.cc:654] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:37.852594  7662 tablet_bootstrap.cc:492] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:37.852738  7662 ts_tablet_manager.cc:1367] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:16:37.853312  7661 tablet_bootstrap.cc:492] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:37.853458  7661 ts_tablet_manager.cc:1367] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 08:16:37.853708  7662 raft_consensus.cc:348] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.853920  7662 raft_consensus.cc:374] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.854009  7662 raft_consensus.cc:729] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.854229  7662 consensus_queue.cc:260] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.854266  7661 raft_consensus.cc:348] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.854419  7661 raft_consensus.cc:374] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.854555  7661 raft_consensus.cc:729] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.854779  7661 consensus_queue.cc:260] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.855279  7661 ts_tablet_manager.cc:1398] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:37.855844  7662 ts_tablet_manager.cc:1398] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:37.859339  7672 raft_consensus.cc:348] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:37.859759  7672 raft_consensus.cc:374] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:37.859864  7672 raft_consensus.cc:729] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:37.860441  7672 consensus_queue.cc:260] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:37.861744  7672 ts_tablet_manager.cc:1398] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.014s	user 0.011s	sys 0.000s
W20240502 08:16:37.875583  7504 tablet.cc:2369] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:37.918870  7675 raft_consensus.cc:482] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.919162  7675 raft_consensus.cc:504] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.920854  7675 leader_election.cc:288] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
W20240502 08:16:37.923722  7373 tablet.cc:2369] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:37.924885  7666 raft_consensus.cc:482] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.925155  7666 raft_consensus.cc:504] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.927363  7666 leader_election.cc:288] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:37.931399  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c06a7143db48fd98fa731b0baa161a" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:37.931900  7196 raft_consensus.cc:2454] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:37.931944  7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c06a7143db48fd98fa731b0baa161a" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:37.932379  7458 raft_consensus.cc:2454] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:37.932591  6866 leader_election.cc:302] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:37.932999  7675 raft_consensus.cc:2781] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.933152  7675 raft_consensus.cc:482] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.933260  7675 raft_consensus.cc:3037] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.936390  7675 raft_consensus.cc:504] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.937173  7675 leader_election.cc:288] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:37.937376  7675 raft_consensus.cc:482] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:37.937516  7675 raft_consensus.cc:504] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.937548  7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c06a7143db48fd98fa731b0baa161a" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:37.937745  7458 raft_consensus.cc:3037] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.938234  7675 leader_election.cc:288] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:37.938745  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c06a7143db48fd98fa731b0baa161a" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:37.938966  7196 raft_consensus.cc:3037] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.940899  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c226114065e6429b8fef55539c315388" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:37.941032  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c06a7143db48fd98fa731b0baa161a" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:37.941255  7457 raft_consensus.cc:2454] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:37.941928  6867 leader_election.cc:302] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:37.942324  7196 raft_consensus.cc:2454] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:37.942718  6866 leader_election.cc:302] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:37.943148  7684 raft_consensus.cc:2781] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:37.943276  7684 raft_consensus.cc:482] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:37.943428  7684 raft_consensus.cc:3037] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.944669  7458 raft_consensus.cc:2454] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:37.945590  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44c06a7143db48fd98fa731b0baa161a" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:37.945781  7675 raft_consensus.cc:2781] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.946493  7675 raft_consensus.cc:686] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [term 1 LEADER]: Becoming Leader. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Running, Role: LEADER
I20240502 08:16:37.947139  7684 raft_consensus.cc:504] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.947075  7675 consensus_queue.cc:237] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.947409  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c226114065e6429b8fef55539c315388" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:37.947932  7327 raft_consensus.cc:2454] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:37.948045  7684 leader_election.cc:288] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:37.948151  7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c226114065e6429b8fef55539c315388" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:37.948576  7458 raft_consensus.cc:3037] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.948678  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c226114065e6429b8fef55539c315388" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:37.948848  7327 raft_consensus.cc:3037] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:37.951427  7458 raft_consensus.cc:2454] T c226114065e6429b8fef55539c315388 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:37.951831  7327 raft_consensus.cc:2454] T c226114065e6429b8fef55539c315388 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:37.951851  6867 leader_election.cc:302] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:37.952109  7675 raft_consensus.cc:2781] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:37.951999  7391 leader_election.cc:302] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f; no voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b
I20240502 08:16:37.952358  7675 raft_consensus.cc:686] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [term 1 LEADER]: Becoming Leader. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Running, Role: LEADER
I20240502 08:16:37.952459  7666 raft_consensus.cc:2726] T 44c06a7143db48fd98fa731b0baa161a P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:37.952625  7675 consensus_queue.cc:237] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:37.954813  6791 catalog_manager.cc:5484] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc93fe73601544fe8a8aa013f8de095b (127.6.151.65). New cstate: current_term: 1 leader_uuid: "cc93fe73601544fe8a8aa013f8de095b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:37.960032  6791 catalog_manager.cc:5484] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc93fe73601544fe8a8aa013f8de095b (127.6.151.65). New cstate: current_term: 1 leader_uuid: "cc93fe73601544fe8a8aa013f8de095b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: HEALTHY } } }
W20240502 08:16:37.961194  7242 tablet.cc:2369] T 44c06a7143db48fd98fa731b0baa161a P 136680942cc947b0b19be2f50bb114eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:38.007388  7111 tablet.cc:2369] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:16:38.060039  6979 tablet.cc:2369] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:16:38.067569  7669 raft_consensus.cc:482] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:38.067857  7669 raft_consensus.cc:504] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:38.069602  7669 leader_election.cc:288] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:38.076539  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7de5d2bd3b4518b1c4d7389f7c5fc0" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:38.076798  7196 raft_consensus.cc:2454] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 0.
I20240502 08:16:38.076761  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7de5d2bd3b4518b1c4d7389f7c5fc0" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:38.077279  7065 raft_consensus.cc:2454] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 0.
I20240502 08:16:38.077302  7522 leader_election.cc:302] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:38.077731  7669 raft_consensus.cc:2781] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:38.077889  7669 raft_consensus.cc:482] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:38.077993  7669 raft_consensus.cc:3037] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.080001  7669 raft_consensus.cc:504] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:38.080518  7669 leader_election.cc:288] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Requested vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:38.080968  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7de5d2bd3b4518b1c4d7389f7c5fc0" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:38.081144  7196 raft_consensus.cc:3037] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.082962  7196 raft_consensus.cc:2454] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:38.083240  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc7de5d2bd3b4518b1c4d7389f7c5fc0" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:38.083324  7522 leader_election.cc:302] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:38.083457  7065 raft_consensus.cc:3037] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:38.083591  7669 raft_consensus.cc:2781] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:38.084048  7669 raft_consensus.cc:686] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [term 1 LEADER]: Becoming Leader. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Running, Role: LEADER
I20240502 08:16:38.084602  7669 consensus_queue.cc:237] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:38.085817  7065 raft_consensus.cc:2454] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:38.089144  6791 catalog_manager.cc:5484] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 881e59a197394322ade814528d5e2cdf (127.6.151.70). New cstate: current_term: 1 leader_uuid: "881e59a197394322ade814528d5e2cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:38.396797  7675 consensus_queue.cc:1035] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.407272  7685 consensus_queue.cc:1035] T c226114065e6429b8fef55539c315388 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.407589  7684 consensus_queue.cc:1035] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.419304  7675 consensus_queue.cc:1035] T 44c06a7143db48fd98fa731b0baa161a P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:38.497184  7692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:38.497512  7692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:38.505278  7692 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 08:16:38.544454  7669 consensus_queue.cc:1035] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:38.552131  7669 consensus_queue.cc:1035] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:38.861084  6867 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:44083, user_credentials={real_user=slave}} blocked reactor thread for 73217.6us

INSERT report
    rows total: 2000
    time total: 291.65 ms
  time per row: 0.145825 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:16:39.915715  6787 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:39.927687  7045 tablet_service.cc:1467] Processing CreateTablet for tablet 6331a099757442eca1206f464bb354a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.927649  7307 tablet_service.cc:1467] Processing CreateTablet for tablet 6331a099757442eca1206f464bb354a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.928135  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6331a099757442eca1206f464bb354a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.928136  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6331a099757442eca1206f464bb354a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.929847  6912 tablet_service.cc:1467] Processing CreateTablet for tablet 030c13ba7de146589cd58a8d3ea774de (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.930248  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030c13ba7de146589cd58a8d3ea774de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.930279  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 030c13ba7de146589cd58a8d3ea774de (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.930665  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030c13ba7de146589cd58a8d3ea774de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.929224  7176 tablet_service.cc:1467] Processing CreateTablet for tablet 030c13ba7de146589cd58a8d3ea774de (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.931082  7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030c13ba7de146589cd58a8d3ea774de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.934576  7569 tablet_service.cc:1467] Processing CreateTablet for tablet 6331a099757442eca1206f464bb354a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.935029  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6331a099757442eca1206f464bb354a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.938134  7768 tablet_bootstrap.cc:492] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:39.939149  7044 tablet_service.cc:1467] Processing CreateTablet for tablet 2de7d4d111364e048241d0f8b2832241 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.939563  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de7d4d111364e048241d0f8b2832241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.939909  7768 tablet_bootstrap.cc:654] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.941247  7769 tablet_bootstrap.cc:492] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:39.942874  7769 tablet_bootstrap.cc:654] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.943085  7568 tablet_service.cc:1467] Processing CreateTablet for tablet 2de7d4d111364e048241d0f8b2832241 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.943509  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de7d4d111364e048241d0f8b2832241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.948091  7307 tablet_service.cc:1467] Processing CreateTablet for tablet 2de7d4d111364e048241d0f8b2832241 (DEFAULT_TABLE table=rebalance_test_table_1 [id=44fd70e7a5814669b511f699ff4fb859]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:39.948514  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de7d4d111364e048241d0f8b2832241. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:39.955245  7769 tablet_bootstrap.cc:492] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:39.955440  7769 ts_tablet_manager.cc:1367] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.014s	user 0.000s	sys 0.005s
I20240502 08:16:39.956041  7774 tablet_bootstrap.cc:492] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:39.956465  7769 raft_consensus.cc:348] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.956683  7769 raft_consensus.cc:374] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.956760  7769 raft_consensus.cc:729] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.957007  7769 consensus_queue.cc:260] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.958489  7773 tablet_bootstrap.cc:492] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:39.960083  7774 tablet_bootstrap.cc:654] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.960988  7769 ts_tablet_manager.cc:1398] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 08:16:39.961370  7769 tablet_bootstrap.cc:492] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:39.963131  7769 tablet_bootstrap.cc:654] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.964943  7773 tablet_bootstrap.cc:654] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.967408  7774 tablet_bootstrap.cc:492] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:39.967695  7774 ts_tablet_manager.cc:1367] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.000s
I20240502 08:16:39.968191  7779 tablet_bootstrap.cc:492] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:39.968763  7774 raft_consensus.cc:348] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:39.969513  7770 tablet_bootstrap.cc:492] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:39.969944  7779 tablet_bootstrap.cc:654] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.970749  7774 raft_consensus.cc:374] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.970921  7770 tablet_bootstrap.cc:654] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.971760  7773 tablet_bootstrap.cc:492] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:39.971920  7773 ts_tablet_manager.cc:1367] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.014s	user 0.004s	sys 0.000s
I20240502 08:16:39.972889  7773 raft_consensus.cc:348] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:39.973193  7773 raft_consensus.cc:374] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.973290  7773 raft_consensus.cc:729] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.973646  7768 tablet_bootstrap.cc:492] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:39.973562  7773 consensus_queue.cc:260] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:39.973850  7768 ts_tablet_manager.cc:1367] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.036s	user 0.000s	sys 0.010s
I20240502 08:16:39.973886  7774 raft_consensus.cc:729] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.974217  7774 consensus_queue.cc:260] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:39.975198  7773 ts_tablet_manager.cc:1398] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:16:39.975139  7768 raft_consensus.cc:348] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.975363  7768 raft_consensus.cc:374] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.975494  7768 raft_consensus.cc:729] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.975636  7769 tablet_bootstrap.cc:492] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:39.975821  7769 ts_tablet_manager.cc:1367] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.001s
I20240502 08:16:39.975744  7768 consensus_queue.cc:260] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.976802  7769 raft_consensus.cc:348] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.977161  7769 raft_consensus.cc:374] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.977423  7769 raft_consensus.cc:729] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.978024  7769 consensus_queue.cc:260] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.980454  7769 ts_tablet_manager.cc:1398] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 08:16:39.985577  7770 tablet_bootstrap.cc:492] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:39.985875  7770 ts_tablet_manager.cc:1367] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.002s
I20240502 08:16:39.987004  7770 raft_consensus.cc:348] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:39.987386  7770 raft_consensus.cc:374] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.987543  7770 raft_consensus.cc:729] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.987951  7770 consensus_queue.cc:260] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:39.988755  7768 ts_tablet_manager.cc:1398] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.004s
I20240502 08:16:39.989043  7774 ts_tablet_manager.cc:1398] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.021s	user 0.012s	sys 0.003s
I20240502 08:16:39.989104  7768 tablet_bootstrap.cc:492] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:39.991410  7768 tablet_bootstrap.cc:654] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:39.993155  7770 ts_tablet_manager.cc:1398] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.002s
I20240502 08:16:39.994741  7779 tablet_bootstrap.cc:492] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:39.994964  7779 ts_tablet_manager.cc:1367] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.003s
I20240502 08:16:39.995698  7768 tablet_bootstrap.cc:492] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:39.995858  7768 ts_tablet_manager.cc:1367] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.007s	user 0.005s	sys 0.000s
I20240502 08:16:39.996001  7779 raft_consensus.cc:348] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.996214  7779 raft_consensus.cc:374] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.996311  7779 raft_consensus.cc:729] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.996546  7779 consensus_queue.cc:260] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.996613  7768 raft_consensus.cc:348] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.996835  7768 raft_consensus.cc:374] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:39.996953  7768 raft_consensus.cc:729] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:39.997234  7779 ts_tablet_manager.cc:1398] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:39.997206  7768 consensus_queue.cc:260] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:39.997538  7779 tablet_bootstrap.cc:492] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:39.997772  7768 ts_tablet_manager.cc:1398] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:39.998982  7779 tablet_bootstrap.cc:654] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:40.000794  7779 tablet_bootstrap.cc:492] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:40.001034  7779 ts_tablet_manager.cc:1367] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:16:40.001732  7779 raft_consensus.cc:348] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.001900  7779 raft_consensus.cc:374] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:40.001993  7779 raft_consensus.cc:729] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:40.002207  7779 consensus_queue.cc:260] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.002676  7779 ts_tablet_manager.cc:1398] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:16:40.124652  7744 raft_consensus.cc:482] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.124929  7744 raft_consensus.cc:504] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:40.125764  7744 leader_election.cc:288] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083)
I20240502 08:16:40.126158  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030c13ba7de146589cd58a8d3ea774de" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:40.126256  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030c13ba7de146589cd58a8d3ea774de" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:40.126356  7196 raft_consensus.cc:2454] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:40.126482  7457 raft_consensus.cc:2454] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:40.126833  6866 leader_election.cc:302] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:40.127122  7744 raft_consensus.cc:2781] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:40.127215  7744 raft_consensus.cc:482] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:40.127300  7744 raft_consensus.cc:3037] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.129088  7744 raft_consensus.cc:504] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:40.129609  7744 leader_election.cc:288] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Requested vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083)
I20240502 08:16:40.130103  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030c13ba7de146589cd58a8d3ea774de" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:40.130112  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030c13ba7de146589cd58a8d3ea774de" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:40.130303  7457 raft_consensus.cc:3037] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.130296  7196 raft_consensus.cc:3037] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.133064  7457 raft_consensus.cc:2454] T 030c13ba7de146589cd58a8d3ea774de P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:40.133064  7196 raft_consensus.cc:2454] T 030c13ba7de146589cd58a8d3ea774de P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:40.133469  6867 leader_election.cc:302] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:40.133751  7744 raft_consensus.cc:2781] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:40.134075  7744 raft_consensus.cc:686] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [term 1 LEADER]: Becoming Leader. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Running, Role: LEADER
I20240502 08:16:40.134387  7744 consensus_queue.cc:237] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:40.137029  6784 catalog_manager.cc:5484] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc93fe73601544fe8a8aa013f8de095b (127.6.151.65). New cstate: current_term: 1 leader_uuid: "cc93fe73601544fe8a8aa013f8de095b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:40.152120  7690 raft_consensus.cc:482] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.152283  7690 raft_consensus.cc:504] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.153012  7690 leader_election.cc:288] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:40.153616  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6331a099757442eca1206f464bb354a5" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:40.153890  7064 raft_consensus.cc:2454] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 0.
I20240502 08:16:40.154405  7524 leader_election.cc:302] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:40.154712  7690 raft_consensus.cc:2781] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:40.154865  7690 raft_consensus.cc:482] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:40.154976  7690 raft_consensus.cc:3037] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.157668  7690 raft_consensus.cc:504] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.158306  7690 leader_election.cc:288] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:40.158782  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6331a099757442eca1206f464bb354a5" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:40.159020  7064 raft_consensus.cc:3037] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.160545  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6331a099757442eca1206f464bb354a5" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:40.160491  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6331a099757442eca1206f464bb354a5" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:40.160766  7327 raft_consensus.cc:3037] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.161689  7064 raft_consensus.cc:2454] T 6331a099757442eca1206f464bb354a5 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:40.162068  7524 leader_election.cc:302] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:40.162317  7690 raft_consensus.cc:2781] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:40.162710  7690 raft_consensus.cc:686] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [term 1 LEADER]: Becoming Leader. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Running, Role: LEADER
I20240502 08:16:40.163064  7327 raft_consensus.cc:2454] T 6331a099757442eca1206f464bb354a5 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:40.163050  7690 consensus_queue.cc:237] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.165861  6784 catalog_manager.cc:5484] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 881e59a197394322ade814528d5e2cdf (127.6.151.70). New cstate: current_term: 1 leader_uuid: "881e59a197394322ade814528d5e2cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:40.210155  7783 raft_consensus.cc:482] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:40.210345  7783 raft_consensus.cc:504] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.211915  7783 leader_election.cc:288] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:40.220140  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de7d4d111364e048241d0f8b2832241" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:40.220391  7327 raft_consensus.cc:2454] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:40.220646  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de7d4d111364e048241d0f8b2832241" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:40.220845  7001 leader_election.cc:302] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:40.220970  7589 raft_consensus.cc:2454] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:40.221277  7783 raft_consensus.cc:2781] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:40.221387  7783 raft_consensus.cc:482] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:40.221511  7783 raft_consensus.cc:3037] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.223553  7783 raft_consensus.cc:504] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.224177  7783 leader_election.cc:288] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Requested vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:40.224604  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de7d4d111364e048241d0f8b2832241" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:40.224802  7327 raft_consensus.cc:3037] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.224761  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de7d4d111364e048241d0f8b2832241" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:40.224925  7589 raft_consensus.cc:3037] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:40.226676  7327 raft_consensus.cc:2454] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:40.226840  7589 raft_consensus.cc:2454] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:40.227100  7001 leader_election.cc:302] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:40.227363  7783 raft_consensus.cc:2781] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:40.227831  7783 raft_consensus.cc:686] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [term 1 LEADER]: Becoming Leader. State: Replica: b8432324fa494be3a41163adc25b0865, State: Running, Role: LEADER
I20240502 08:16:40.228497  7783 consensus_queue.cc:237] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:40.231860  6784 catalog_manager.cc:5484] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8432324fa494be3a41163adc25b0865 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "b8432324fa494be3a41163adc25b0865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
W20240502 08:16:40.578053  7798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:40.578410  7798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:40.586355  7798 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:40.608871  7744 consensus_queue.cc:1035] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.615969  7787 consensus_queue.cc:1035] T 030c13ba7de146589cd58a8d3ea774de P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 08:16:40.662016  7690 consensus_queue.cc:1035] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.669098  7831 consensus_queue.cc:1035] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:40.732837  7327 raft_consensus.cc:1261] T 2de7d4d111364e048241d0f8b2832241 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Refusing update from remote peer b8432324fa494be3a41163adc25b0865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:40.732990  7589 raft_consensus.cc:1261] T 2de7d4d111364e048241d0f8b2832241 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Refusing update from remote peer b8432324fa494be3a41163adc25b0865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:16:40.733577  7783 consensus_queue.cc:1035] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240502 08:16:40.736842  7783 consensus_queue.cc:1035] T 2de7d4d111364e048241d0f8b2832241 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:40.790685  7000 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:36909, user_credentials={real_user=slave}} blocked reactor thread for 56981.8us
I20240502 08:16:40.851128  7854 mvcc.cc:205] Tried to move back new op lower bound from 7023156431783219200 to 7023156429739773952. Current Snapshot: MvccSnapshot[applied={T|T < 7023156431783219200}]

INSERT report
    rows total: 2000
    time total: 253.713 ms
  time per row: 0.126856 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      4       1192 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:16:41.937134  6784 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:41.946166  6912 tablet_service.cc:1467] Processing CreateTablet for tablet aec8be7dfe664e29ad5153c154d4f585 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.946532  7437 tablet_service.cc:1467] Processing CreateTablet for tablet aec8be7dfe664e29ad5153c154d4f585 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.946703  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec8be7dfe664e29ad5153c154d4f585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.947005  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec8be7dfe664e29ad5153c154d4f585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.948046  7307 tablet_service.cc:1467] Processing CreateTablet for tablet a27e8d56086e44a99aade2205fa32f18 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.948472  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27e8d56086e44a99aade2205fa32f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.951076  7176 tablet_service.cc:1467] Processing CreateTablet for tablet aec8be7dfe664e29ad5153c154d4f585 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.951493  7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aec8be7dfe664e29ad5153c154d4f585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.951704  7044 tablet_service.cc:1467] Processing CreateTablet for tablet a27e8d56086e44a99aade2205fa32f18 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.952133  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27e8d56086e44a99aade2205fa32f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.952210  7438 tablet_service.cc:1467] Processing CreateTablet for tablet 62c40420104d40e6853ea6fbbf9db21a (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.952576  7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c40420104d40e6853ea6fbbf9db21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.955093  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 62c40420104d40e6853ea6fbbf9db21a (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.955173  7569 tablet_service.cc:1467] Processing CreateTablet for tablet a27e8d56086e44a99aade2205fa32f18 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.955457  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c40420104d40e6853ea6fbbf9db21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.955710  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a27e8d56086e44a99aade2205fa32f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.959702  6913 tablet_service.cc:1467] Processing CreateTablet for tablet 62c40420104d40e6853ea6fbbf9db21a (DEFAULT_TABLE table=rebalance_test_table_2 [id=3b714707f91942ccb444d0e0cbc60461]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:41.960139  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c40420104d40e6853ea6fbbf9db21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:41.961781  7874 tablet_bootstrap.cc:492] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:41.963449  7874 tablet_bootstrap.cc:654] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.973160  7874 tablet_bootstrap.cc:492] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:41.973388  7874 ts_tablet_manager.cc:1367] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.005s
I20240502 08:16:41.974292  7874 raft_consensus.cc:348] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:41.974679  7874 raft_consensus.cc:374] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.974838  7874 raft_consensus.cc:729] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.975185  7874 consensus_queue.cc:260] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:41.975912  7879 tablet_bootstrap.cc:492] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:41.977330  7877 tablet_bootstrap.cc:492] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:41.978829  7879 tablet_bootstrap.cc:654] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.979081  7877 tablet_bootstrap.cc:654] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.982460  7874 ts_tablet_manager.cc:1398] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20240502 08:16:41.983074  7874 tablet_bootstrap.cc:492] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:41.986071  7874 tablet_bootstrap.cc:654] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.987632  7880 tablet_bootstrap.cc:492] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:41.989737  7880 tablet_bootstrap.cc:654] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.991235  7877 tablet_bootstrap.cc:492] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:41.991464  7877 ts_tablet_manager.cc:1367] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.014s	user 0.006s	sys 0.001s
I20240502 08:16:41.992353  7878 tablet_bootstrap.cc:492] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:41.992601  7877 raft_consensus.cc:348] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:41.992906  7877 raft_consensus.cc:374] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:41.993034  7877 raft_consensus.cc:729] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:41.993384  7877 consensus_queue.cc:260] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:41.994282  7878 tablet_bootstrap.cc:654] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.995517  7875 tablet_bootstrap.cc:492] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:41.995913  7877 ts_tablet_manager.cc:1398] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 08:16:41.996289  7877 tablet_bootstrap.cc:492] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:41.997689  7875 tablet_bootstrap.cc:654] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:41.998442  7877 tablet_bootstrap.cc:654] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.000113  7880 tablet_bootstrap.cc:492] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:42.000265  7880 ts_tablet_manager.cc:1367] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.000s
I20240502 08:16:42.000555  7874 tablet_bootstrap.cc:492] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:42.000725  7874 ts_tablet_manager.cc:1367] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.000s
I20240502 08:16:42.001133  7880 raft_consensus.cc:348] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.001452  7874 raft_consensus.cc:348] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:42.001708  7874 raft_consensus.cc:374] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.001788  7874 raft_consensus.cc:729] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.002048  7874 consensus_queue.cc:260] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:42.002408  7875 tablet_bootstrap.cc:492] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:42.002513  7880 raft_consensus.cc:374] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.002601  7874 ts_tablet_manager.cc:1398] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:42.002818  7875 ts_tablet_manager.cc:1367] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 08:16:42.002631  7880 raft_consensus.cc:729] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.003829  7875 raft_consensus.cc:348] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.003995  7875 raft_consensus.cc:374] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.004093  7875 raft_consensus.cc:729] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.004354  7880 consensus_queue.cc:260] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.004966  7880 ts_tablet_manager.cc:1398] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:16:42.005854  7875 consensus_queue.cc:260] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.006348  7879 tablet_bootstrap.cc:492] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:42.006552  7875 ts_tablet_manager.cc:1398] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:42.006708  7879 ts_tablet_manager.cc:1367] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.031s	user 0.009s	sys 0.019s
I20240502 08:16:42.008184  7879 raft_consensus.cc:348] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:42.008458  7879 raft_consensus.cc:374] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.008582  7879 raft_consensus.cc:729] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.008931  7879 consensus_queue.cc:260] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:42.014312  7879 ts_tablet_manager.cc:1398] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.007s	user 0.000s	sys 0.006s
I20240502 08:16:42.014700  7879 tablet_bootstrap.cc:492] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:42.015833  7869 raft_consensus.cc:482] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.015981  7869 raft_consensus.cc:504] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.016537  7879 tablet_bootstrap.cc:654] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:42.016656  7869 leader_election.cc:288] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:42.017634  7878 tablet_bootstrap.cc:492] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:42.017645  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27e8d56086e44a99aade2205fa32f18" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:42.017809  7878 ts_tablet_manager.cc:1367] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.026s	user 0.003s	sys 0.000s
I20240502 08:16:42.018375  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27e8d56086e44a99aade2205fa32f18" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:42.018653  7878 raft_consensus.cc:348] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.018852  7878 raft_consensus.cc:374] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.018944  7878 raft_consensus.cc:729] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.017874  7589 raft_consensus.cc:2454] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:42.019165  7878 consensus_queue.cc:260] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.019831  7000 leader_election.cc:302] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
W20240502 08:16:42.021919  7001 leader_election.cc:341] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:42.023782  7877 tablet_bootstrap.cc:492] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:42.024605  7877 ts_tablet_manager.cc:1367] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.001s
I20240502 08:16:42.024935  7869 raft_consensus.cc:2781] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:42.025187  7869 raft_consensus.cc:482] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:42.025326  7869 raft_consensus.cc:3037] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.026118  7877 raft_consensus.cc:348] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:42.027269  7877 raft_consensus.cc:374] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.027556  7877 raft_consensus.cc:729] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.028038  7877 consensus_queue.cc:260] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:42.028671  7869 raft_consensus.cc:504] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.029258  7877 ts_tablet_manager.cc:1398] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 08:16:42.030406  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27e8d56086e44a99aade2205fa32f18" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:42.030668  7589 raft_consensus.cc:3037] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.032305  7878 ts_tablet_manager.cc:1398] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.003s
I20240502 08:16:42.033459  7589 raft_consensus.cc:2454] T a27e8d56086e44a99aade2205fa32f18 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:42.033927  7000 leader_election.cc:302] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:42.041214  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a27e8d56086e44a99aade2205fa32f18" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:42.041481  7327 raft_consensus.cc:3037] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.043109  7869 leader_election.cc:288] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Requested vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:42.043335  7869 raft_consensus.cc:2781] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:42.043445  7869 raft_consensus.cc:686] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [term 1 LEADER]: Becoming Leader. State: Replica: b8432324fa494be3a41163adc25b0865, State: Running, Role: LEADER
I20240502 08:16:42.044050  7879 tablet_bootstrap.cc:492] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:42.044306  7879 ts_tablet_manager.cc:1367] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.017s
I20240502 08:16:42.044304  7327 raft_consensus.cc:2454] T a27e8d56086e44a99aade2205fa32f18 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:42.044275  7869 consensus_queue.cc:237] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:42.045045  7879 raft_consensus.cc:348] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:42.045219  7879 raft_consensus.cc:374] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:42.045291  7879 raft_consensus.cc:729] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:42.045542  7879 consensus_queue.cc:260] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:42.046828  7879 ts_tablet_manager.cc:1398] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.002s
I20240502 08:16:42.048934  6784 catalog_manager.cc:5484] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8432324fa494be3a41163adc25b0865 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "b8432324fa494be3a41163adc25b0865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:42.078115  7889 raft_consensus.cc:482] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.078308  7889 raft_consensus.cc:504] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:42.079980  7889 leader_election.cc:288] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:42.099334  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c40420104d40e6853ea6fbbf9db21a" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:42.099639  7457 raft_consensus.cc:2454] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:42.100169  7130 leader_election.cc:302] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:42.100308  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c40420104d40e6853ea6fbbf9db21a" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:42.100579  7889 raft_consensus.cc:2781] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:42.100592  6933 raft_consensus.cc:2454] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:42.100683  7889 raft_consensus.cc:482] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:42.100772  7889 raft_consensus.cc:3037] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.102807  7889 raft_consensus.cc:504] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:42.103425  7889 leader_election.cc:288] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:42.103806  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c40420104d40e6853ea6fbbf9db21a" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:42.103832  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c40420104d40e6853ea6fbbf9db21a" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:42.103998  6933 raft_consensus.cc:3037] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.104009  7457 raft_consensus.cc:3037] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.105969  6933 raft_consensus.cc:2454] T 62c40420104d40e6853ea6fbbf9db21a P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:42.106158  7457 raft_consensus.cc:2454] T 62c40420104d40e6853ea6fbbf9db21a P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:42.106376  7129 leader_election.cc:302] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:42.106717  7889 raft_consensus.cc:2781] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:42.106887  7889 raft_consensus.cc:686] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [term 1 LEADER]: Becoming Leader. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Running, Role: LEADER
I20240502 08:16:42.107344  7889 consensus_queue.cc:237] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:42.110831  6784 catalog_manager.cc:5484] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 136680942cc947b0b19be2f50bb114eb (127.6.151.67). New cstate: current_term: 1 leader_uuid: "136680942cc947b0b19be2f50bb114eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:42.212796  7881 raft_consensus.cc:482] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:42.213002  7881 raft_consensus.cc:504] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:42.213729  7881 leader_election.cc:288] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:42.214166  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec8be7dfe664e29ad5153c154d4f585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:42.214243  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec8be7dfe664e29ad5153c154d4f585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:42.214359  6933 raft_consensus.cc:2454] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:42.214498  7196 raft_consensus.cc:2454] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:42.214804  7391 leader_election.cc:302] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:42.215068  7881 raft_consensus.cc:2781] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:42.215186  7881 raft_consensus.cc:482] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:42.215276  7881 raft_consensus.cc:3037] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.217363  7881 raft_consensus.cc:504] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:42.217983  7881 leader_election.cc:288] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Requested vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:42.218524  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec8be7dfe664e29ad5153c154d4f585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:42.218582  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aec8be7dfe664e29ad5153c154d4f585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:42.218725  6933 raft_consensus.cc:3037] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.218849  7196 raft_consensus.cc:3037] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:42.220603  6933 raft_consensus.cc:2454] T aec8be7dfe664e29ad5153c154d4f585 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:42.221033  7391 leader_election.cc:302] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:42.221051  7196 raft_consensus.cc:2454] T aec8be7dfe664e29ad5153c154d4f585 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:42.221343  7881 raft_consensus.cc:2781] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:42.221786  7881 raft_consensus.cc:686] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [term 1 LEADER]: Becoming Leader. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Running, Role: LEADER
I20240502 08:16:42.222214  7881 consensus_queue.cc:237] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:42.225683  6784 catalog_manager.cc:5484] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b21a0f245140b293e607e8f554fa1f (127.6.151.69). New cstate: current_term: 1 leader_uuid: "a5b21a0f245140b293e607e8f554fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:42.545166  7892 consensus_queue.cc:1035] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.551384  7889 consensus_queue.cc:1035] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.557543  7889 consensus_queue.cc:1035] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.581956  7904 consensus_queue.cc:1035] T a27e8d56086e44a99aade2205fa32f18 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:42.625720  7902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:42.626047  7902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:42.633713  7902 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:16:42.781607  7881 consensus_queue.cc:1035] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:42.787125  7881 consensus_queue.cc:1035] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:42.883119  7391 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.67:36791, user_credentials={real_user=slave}} blocked reactor thread for 60240.1us

INSERT report
    rows total: 2000
    time total: 331.385 ms
  time per row: 0.165692 ms
I20240502 08:16:43.120966  6783 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:43.130069  7044 tablet_service.cc:1467] Processing CreateTablet for tablet c9282750898d4375a36d4fd0be7008ec (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.130244  7307 tablet_service.cc:1467] Processing CreateTablet for tablet c9282750898d4375a36d4fd0be7008ec (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.130580  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9282750898d4375a36d4fd0be7008ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.130697  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9282750898d4375a36d4fd0be7008ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.131613  7045 tablet_service.cc:1467] Processing CreateTablet for tablet b4b502d33f5c45d2a0d80c3588404021 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.131762  7568 tablet_service.cc:1467] Processing CreateTablet for tablet b4b502d33f5c45d2a0d80c3588404021 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.132004  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b502d33f5c45d2a0d80c3588404021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.132158  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b502d33f5c45d2a0d80c3588404021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.130530  7569 tablet_service.cc:1467] Processing CreateTablet for tablet c9282750898d4375a36d4fd0be7008ec (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.135272  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9282750898d4375a36d4fd0be7008ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.136946  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 1e0ba483f04745cbb3522ece53fb2567 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.137387  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e0ba483f04745cbb3522ece53fb2567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.143271  6913 tablet_service.cc:1467] Processing CreateTablet for tablet 1e0ba483f04745cbb3522ece53fb2567 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.143864  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e0ba483f04745cbb3522ece53fb2567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.146090  7980 tablet_bootstrap.cc:492] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:43.148723  7980 tablet_bootstrap.cc:654] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.149057  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 1e0ba483f04745cbb3522ece53fb2567 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.149443  7982 tablet_bootstrap.cc:492] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:43.149514  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e0ba483f04745cbb3522ece53fb2567. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.149765  7307 tablet_service.cc:1467] Processing CreateTablet for tablet b4b502d33f5c45d2a0d80c3588404021 (DEFAULT_TABLE table=rebalance_test_table_3 [id=f3357ca9e0f8450ebd456b3c74bf480d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:43.150180  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b502d33f5c45d2a0d80c3588404021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:43.151057  7982 tablet_bootstrap.cc:654] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.160387  7980 tablet_bootstrap.cc:492] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:43.160637  7980 ts_tablet_manager.cc:1367] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.003s
I20240502 08:16:43.161737  7980 raft_consensus.cc:348] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.162128  7980 raft_consensus.cc:374] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.162307  7980 raft_consensus.cc:729] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.162669  7980 consensus_queue.cc:260] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.164441  7982 tablet_bootstrap.cc:492] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:43.165156  7982 ts_tablet_manager.cc:1367] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.000s
I20240502 08:16:43.165927  7987 tablet_bootstrap.cc:492] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:43.166546  7988 tablet_bootstrap.cc:492] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:43.166922  7980 ts_tablet_manager.cc:1398] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.005s
I20240502 08:16:43.167493  7980 tablet_bootstrap.cc:492] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:43.167549  7982 raft_consensus.cc:348] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.167846  7982 raft_consensus.cc:374] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.167995  7982 raft_consensus.cc:729] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.168169  7984 tablet_bootstrap.cc:492] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:43.168314  7982 consensus_queue.cc:260] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.169351  7980 tablet_bootstrap.cc:654] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.169674  7982 ts_tablet_manager.cc:1398] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:43.169998  7982 tablet_bootstrap.cc:492] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:43.171622  7982 tablet_bootstrap.cc:654] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.175868  7984 tablet_bootstrap.cc:654] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.177881  7981 tablet_bootstrap.cc:492] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:43.180171  7981 tablet_bootstrap.cc:654] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.180446  7982 tablet_bootstrap.cc:492] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:43.180740  7982 ts_tablet_manager.cc:1367] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.004s
I20240502 08:16:43.181165  7988 tablet_bootstrap.cc:654] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.181690  7982 raft_consensus.cc:348] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.181998  7982 raft_consensus.cc:374] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.182122  7982 raft_consensus.cc:729] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.182571  7982 consensus_queue.cc:260] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.184082  7982 ts_tablet_manager.cc:1398] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:43.185142  7987 tablet_bootstrap.cc:654] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.185812  7984 tablet_bootstrap.cc:492] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:43.185979  7984 ts_tablet_manager.cc:1367] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.019s	user 0.002s	sys 0.003s
I20240502 08:16:43.188174  7981 tablet_bootstrap.cc:492] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:43.188396  7981 ts_tablet_manager.cc:1367] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240502 08:16:43.188396  7980 tablet_bootstrap.cc:492] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:43.188580  7980 ts_tablet_manager.cc:1367] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.021s	user 0.003s	sys 0.013s
I20240502 08:16:43.188647  7984 raft_consensus.cc:348] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.188932  7984 raft_consensus.cc:374] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.188948  7987 tablet_bootstrap.cc:492] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:43.189086  7987 ts_tablet_manager.cc:1367] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.023s	user 0.004s	sys 0.000s
I20240502 08:16:43.189098  7984 raft_consensus.cc:729] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.189283  7981 raft_consensus.cc:348] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.189519  7981 raft_consensus.cc:374] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.189483  7980 raft_consensus.cc:348] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.189649  7981 raft_consensus.cc:729] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.189716  7980 raft_consensus.cc:374] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.189536  7984 consensus_queue.cc:260] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.189806  7980 raft_consensus.cc:729] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.189844  7987 raft_consensus.cc:348] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.190052  7987 raft_consensus.cc:374] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.189987  7981 consensus_queue.cc:260] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.190148  7987 raft_consensus.cc:729] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.190076  7980 consensus_queue.cc:260] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.190546  7981 ts_tablet_manager.cc:1398] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:16:43.190738  7987 consensus_queue.cc:260] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.191241  7980 ts_tablet_manager.cc:1398] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:16:43.191381  7981 tablet_bootstrap.cc:492] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:43.192155  7987 ts_tablet_manager.cc:1398] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:43.193398  7988 tablet_bootstrap.cc:492] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:43.193482  7981 tablet_bootstrap.cc:654] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:43.193562  7988 ts_tablet_manager.cc:1367] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.027s	user 0.005s	sys 0.000s
I20240502 08:16:43.194681  7988 raft_consensus.cc:348] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.194906  7988 raft_consensus.cc:374] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.195055  7988 raft_consensus.cc:729] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.195346  7988 consensus_queue.cc:260] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.196043  7988 ts_tablet_manager.cc:1398] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:16:43.196516  7986 raft_consensus.cc:482] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.196738  7981 tablet_bootstrap.cc:492] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:43.196683  7986 raft_consensus.cc:504] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.196890  7981 ts_tablet_manager.cc:1367] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 08:16:43.197677  7981 raft_consensus.cc:348] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.198064  7981 raft_consensus.cc:374] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:43.198148  7981 raft_consensus.cc:729] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:43.198246  7984 ts_tablet_manager.cc:1398] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.003s
I20240502 08:16:43.198776  7986 leader_election.cc:288] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:43.198418  7981 consensus_queue.cc:260] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.200273  7981 ts_tablet_manager.cc:1398] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:16:43.211836  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b502d33f5c45d2a0d80c3588404021" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:43.212138  7064 raft_consensus.cc:2454] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:43.212702  7262 leader_election.cc:302] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:43.213168  7986 raft_consensus.cc:2781] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:43.213305  7986 raft_consensus.cc:482] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:43.213413  7986 raft_consensus.cc:3037] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.216948  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b502d33f5c45d2a0d80c3588404021" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:43.217156  7589 raft_consensus.cc:2454] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:43.253882  7999 raft_consensus.cc:482] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.254092  7999 raft_consensus.cc:504] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.254981  7999 leader_election.cc:288] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:43.255438  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9282750898d4375a36d4fd0be7008ec" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:43.255695  7064 raft_consensus.cc:2454] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:43.255626  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9282750898d4375a36d4fd0be7008ec" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:43.255829  7589 raft_consensus.cc:2454] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:43.256047  7262 leader_election.cc:302] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:43.256315  7999 raft_consensus.cc:2781] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:43.256433  7999 raft_consensus.cc:482] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:43.256531  7999 raft_consensus.cc:3037] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.278116  7986 raft_consensus.cc:504] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.278899  7986 leader_election.cc:288] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:43.279511  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b502d33f5c45d2a0d80c3588404021" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:43.279532  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b502d33f5c45d2a0d80c3588404021" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:43.279743  7589 raft_consensus.cc:3037] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.279762  7064 raft_consensus.cc:3037] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.281164  7999 raft_consensus.cc:504] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.281895  7999 leader_election.cc:288] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:43.282554  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9282750898d4375a36d4fd0be7008ec" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:43.282817  7065 raft_consensus.cc:3037] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.283264  7588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9282750898d4375a36d4fd0be7008ec" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:43.283464  7588 raft_consensus.cc:3037] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.303965  7949 raft_consensus.cc:482] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:43.304193  7949 raft_consensus.cc:504] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.304929  7949 leader_election.cc:288] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:43.305414  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0ba483f04745cbb3522ece53fb2567" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:43.305495  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0ba483f04745cbb3522ece53fb2567" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:43.305680  7196 raft_consensus.cc:2454] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:43.305774  6932 raft_consensus.cc:2454] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:43.306149  7391 leader_election.cc:302] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:43.306401  7949 raft_consensus.cc:2781] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:43.306501  7949 raft_consensus.cc:482] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:43.306565  7949 raft_consensus.cc:3037] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.376446  7064 raft_consensus.cc:2454] T b4b502d33f5c45d2a0d80c3588404021 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:43.376484  7589 raft_consensus.cc:2454] T b4b502d33f5c45d2a0d80c3588404021 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:43.377084  7949 raft_consensus.cc:504] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.377897  7949 leader_election.cc:288] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Requested vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:43.379076  7588 raft_consensus.cc:2454] T c9282750898d4375a36d4fd0be7008ec P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:43.380909  7196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0ba483f04745cbb3522ece53fb2567" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:43.381157  7196 raft_consensus.cc:3037] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.382449  7065 raft_consensus.cc:2454] T c9282750898d4375a36d4fd0be7008ec P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:43.382942  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e0ba483f04745cbb3522ece53fb2567" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:43.383169  6932 raft_consensus.cc:3037] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:43.386150  7262 leader_election.cc:302] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:43.386493  7262 leader_election.cc:302] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:43.386504  7999 raft_consensus.cc:2781] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:43.386785  7999 raft_consensus.cc:686] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:43.386879  7986 raft_consensus.cc:2781] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:43.386982  7986 raft_consensus.cc:686] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:43.387398  7999 consensus_queue.cc:237] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.387627  7986 consensus_queue.cc:237] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:43.398437  6783 catalog_manager.cc:5484] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:43.403903  6783 catalog_manager.cc:5484] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:43.481657  6932 raft_consensus.cc:2454] T 1e0ba483f04745cbb3522ece53fb2567 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:43.481638  7196 raft_consensus.cc:2454] T 1e0ba483f04745cbb3522ece53fb2567 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:43.482141  7391 leader_election.cc:302] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:43.482439  7949 raft_consensus.cc:2781] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:43.482599  7949 raft_consensus.cc:686] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [term 1 LEADER]: Becoming Leader. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Running, Role: LEADER
I20240502 08:16:43.482975  7949 consensus_queue.cc:237] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:43.486407  6783 catalog_manager.cc:5484] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b21a0f245140b293e607e8f554fa1f (127.6.151.69). New cstate: current_term: 1 leader_uuid: "a5b21a0f245140b293e607e8f554fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:43.835032  7986 consensus_queue.cc:1035] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:43.850030  8000 consensus_queue.cc:1035] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:43.904175  8004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:43.904541  8004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:43.914741  8004 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:43.956527  8000 consensus_queue.cc:1035] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:43.957880  8000 consensus_queue.cc:1035] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 08:16:44.022437  7949 consensus_queue.cc:1035] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:44.027185  7949 consensus_queue.cc:1035] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 283.216 ms
  time per row: 0.141608 ms
I20240502 08:16:44.388327  6790 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:44.399613  7175 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2c1cbd9c14fddb57a7d9c103fa496 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.399794  6913 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2c1cbd9c14fddb57a7d9c103fa496 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.400111  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2c1cbd9c14fddb57a7d9c103fa496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.400223  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2c1cbd9c14fddb57a7d9c103fa496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.400656  7437 tablet_service.cc:1467] Processing CreateTablet for tablet fdc2c1cbd9c14fddb57a7d9c103fa496 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.401046  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdc2c1cbd9c14fddb57a7d9c103fa496. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.403026  7569 tablet_service.cc:1467] Processing CreateTablet for tablet ec2c667c07b64ac987cdba570dfce4db (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.403328  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2c667c07b64ac987cdba570dfce4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.407532  7045 tablet_service.cc:1467] Processing CreateTablet for tablet ec2c667c07b64ac987cdba570dfce4db (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.407954  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2c667c07b64ac987cdba570dfce4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.408366  7438 tablet_service.cc:1467] Processing CreateTablet for tablet 19c8e9eca3de4a959109f870f9299dd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.408835  7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c8e9eca3de4a959109f870f9299dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.411101  7307 tablet_service.cc:1467] Processing CreateTablet for tablet ec2c667c07b64ac987cdba570dfce4db (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.411659  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2c667c07b64ac987cdba570dfce4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.412724  8063 tablet_bootstrap.cc:492] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:44.414413  6912 tablet_service.cc:1467] Processing CreateTablet for tablet 19c8e9eca3de4a959109f870f9299dd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.414849  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c8e9eca3de4a959109f870f9299dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.417889  7176 tablet_service.cc:1467] Processing CreateTablet for tablet 19c8e9eca3de4a959109f870f9299dd4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6acd67800e344930a63891768ad472b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:44.418295  8064 tablet_bootstrap.cc:492] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:44.418329  7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19c8e9eca3de4a959109f870f9299dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:44.419986  8064 tablet_bootstrap.cc:654] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.427910  8064 tablet_bootstrap.cc:492] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:44.428153  8064 ts_tablet_manager.cc:1367] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.003s
I20240502 08:16:44.429030  8064 raft_consensus.cc:348] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.429354  8064 raft_consensus.cc:374] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.429494  8064 raft_consensus.cc:729] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.429826  8064 consensus_queue.cc:260] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.430563  8064 ts_tablet_manager.cc:1398] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:16:44.430907  8064 tablet_bootstrap.cc:492] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:44.432380  8069 tablet_bootstrap.cc:492] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:44.432585  8064 tablet_bootstrap.cc:654] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.433488  8067 tablet_bootstrap.cc:492] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:44.434047  8068 tablet_bootstrap.cc:492] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:44.435875  8068 tablet_bootstrap.cc:654] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.436235  8069 tablet_bootstrap.cc:654] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.438406  8063 tablet_bootstrap.cc:654] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.438719  8065 tablet_bootstrap.cc:492] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:44.440009  8064 tablet_bootstrap.cc:492] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:44.440224  8064 ts_tablet_manager.cc:1367] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.008s
I20240502 08:16:44.440310  8067 tablet_bootstrap.cc:654] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.440529  8065 tablet_bootstrap.cc:654] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.440523  8068 tablet_bootstrap.cc:492] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:44.440670  8068 ts_tablet_manager.cc:1367] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 08:16:44.441102  8064 raft_consensus.cc:348] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.441395  8064 raft_consensus.cc:374] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.441524  8064 raft_consensus.cc:729] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.441480  8068 raft_consensus.cc:348] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.441763  8068 raft_consensus.cc:374] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.441890  8068 raft_consensus.cc:729] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.442026  8064 consensus_queue.cc:260] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.443506  8064 ts_tablet_manager.cc:1398] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:16:44.442152  8068 consensus_queue.cc:260] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.444825  8063 tablet_bootstrap.cc:492] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:44.445582  8063 ts_tablet_manager.cc:1367] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.033s	user 0.001s	sys 0.004s
I20240502 08:16:44.446339  8068 ts_tablet_manager.cc:1398] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 08:16:44.446419  8063 raft_consensus.cc:348] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.447331  8069 tablet_bootstrap.cc:492] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:44.447532  8069 ts_tablet_manager.cc:1367] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.000s
I20240502 08:16:44.448077  8063 raft_consensus.cc:374] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.448217  8063 raft_consensus.cc:729] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.448379  8069 raft_consensus.cc:348] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.448659  8069 raft_consensus.cc:374] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.448809  8069 raft_consensus.cc:729] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.449040  8067 tablet_bootstrap.cc:492] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:44.449067  8069 consensus_queue.cc:260] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.449221  8067 ts_tablet_manager.cc:1367] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.016s	user 0.000s	sys 0.004s
I20240502 08:16:44.449690  8069 ts_tablet_manager.cc:1398] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:44.448590  8063 consensus_queue.cc:260] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.449836  8065 tablet_bootstrap.cc:492] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:44.450022  8065 ts_tablet_manager.cc:1367] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.003s
I20240502 08:16:44.450245  8063 ts_tablet_manager.cc:1398] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 08:16:44.450552  8063 tablet_bootstrap.cc:492] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:44.451128  8065 raft_consensus.cc:348] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.451876  8067 raft_consensus.cc:348] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.452123  8065 raft_consensus.cc:374] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.452255  8065 raft_consensus.cc:729] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.452478  8065 consensus_queue.cc:260] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.453027  8065 ts_tablet_manager.cc:1398] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:16:44.453167  8067 raft_consensus.cc:374] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.453260  8067 raft_consensus.cc:729] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.453495  8067 consensus_queue.cc:260] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.453680  8063 tablet_bootstrap.cc:654] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.453974  8065 tablet_bootstrap.cc:492] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:44.454336  8067 ts_tablet_manager.cc:1398] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 08:16:44.456514  8065 tablet_bootstrap.cc:654] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:44.460117  8058 raft_consensus.cc:482] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.460301  8058 raft_consensus.cc:504] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.461254  8058 leader_election.cc:288] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:44.462193  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2c667c07b64ac987cdba570dfce4db" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:44.462502  7327 raft_consensus.cc:2454] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 0.
I20240502 08:16:44.463318  7525 leader_election.cc:302] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:44.464324  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2c667c07b64ac987cdba570dfce4db" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:44.464504  8058 raft_consensus.cc:2781] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:44.464560  7064 raft_consensus.cc:2454] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 0.
I20240502 08:16:44.464624  8058 raft_consensus.cc:482] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:44.464704  8058 raft_consensus.cc:3037] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.476023  8065 tablet_bootstrap.cc:492] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:44.476212  8065 ts_tablet_manager.cc:1367] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.009s
I20240502 08:16:44.476495  8063 tablet_bootstrap.cc:492] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:44.476666  8063 ts_tablet_manager.cc:1367] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.026s	user 0.006s	sys 0.011s
I20240502 08:16:44.476933  8065 raft_consensus.cc:348] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.477152  8065 raft_consensus.cc:374] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.477236  8065 raft_consensus.cc:729] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.477422  8063 raft_consensus.cc:348] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.477499  8065 consensus_queue.cc:260] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.477640  8063 raft_consensus.cc:374] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:44.477720  8063 raft_consensus.cc:729] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:44.478109  8063 consensus_queue.cc:260] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.478335  8036 raft_consensus.cc:482] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.478440  8036 raft_consensus.cc:504] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.478749  8065 ts_tablet_manager.cc:1398] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:16:44.479209  8036 leader_election.cc:288] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:44.479713  8063 ts_tablet_manager.cc:1398] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.002s
I20240502 08:16:44.479966  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8e9eca3de4a959109f870f9299dd4" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:44.480024  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8e9eca3de4a959109f870f9299dd4" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:44.480180  7195 raft_consensus.cc:2454] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:44.480238  6932 raft_consensus.cc:2454] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:44.480625  7391 leader_election.cc:302] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:44.480883  8036 raft_consensus.cc:2781] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:44.481009  8036 raft_consensus.cc:482] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:44.481107  8036 raft_consensus.cc:3037] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.492007  8036 raft_consensus.cc:504] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.492007  8058 raft_consensus.cc:504] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.492630  8036 leader_election.cc:288] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Requested vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:44.493083  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8e9eca3de4a959109f870f9299dd4" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:44.493109  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2c667c07b64ac987cdba570dfce4db" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:44.493192  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19c8e9eca3de4a959109f870f9299dd4" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:44.493307  7064 raft_consensus.cc:3037] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.493328  6932 raft_consensus.cc:3037] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.493409  7195 raft_consensus.cc:3037] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.496428  8058 leader_election.cc:288] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:44.496802  7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2c667c07b64ac987cdba570dfce4db" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:44.497009  7327 raft_consensus.cc:3037] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.596581  7195 raft_consensus.cc:2454] T 19c8e9eca3de4a959109f870f9299dd4 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:44.596581  6932 raft_consensus.cc:2454] T 19c8e9eca3de4a959109f870f9299dd4 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:44.596637  7064 raft_consensus.cc:2454] T ec2c667c07b64ac987cdba570dfce4db P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:44.597136  7391 leader_election.cc:302] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:44.597244  7327 raft_consensus.cc:2454] T ec2c667c07b64ac987cdba570dfce4db P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:44.597345  7524 leader_election.cc:302] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:44.597710  8036 raft_consensus.cc:2781] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:44.597867  8036 raft_consensus.cc:686] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [term 1 LEADER]: Becoming Leader. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Running, Role: LEADER
I20240502 08:16:44.598102  8058 raft_consensus.cc:2781] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:44.598165  8036 consensus_queue.cc:237] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:44.598629  8058 raft_consensus.cc:686] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [term 1 LEADER]: Becoming Leader. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Running, Role: LEADER
I20240502 08:16:44.598976  8058 consensus_queue.cc:237] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:44.601663  6790 catalog_manager.cc:5484] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b21a0f245140b293e607e8f554fa1f (127.6.151.69). New cstate: current_term: 1 leader_uuid: "a5b21a0f245140b293e607e8f554fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:44.603910  6785 catalog_manager.cc:5484] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 881e59a197394322ade814528d5e2cdf (127.6.151.70). New cstate: current_term: 1 leader_uuid: "881e59a197394322ade814528d5e2cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:44.815758  7937 raft_consensus.cc:482] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:44.815948  7937 raft_consensus.cc:504] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.816617  7937 leader_election.cc:288] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083)
I20240502 08:16:44.817184  7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2c1cbd9c14fddb57a7d9c103fa496" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:44.817163  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2c1cbd9c14fddb57a7d9c103fa496" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:44.817442  7195 raft_consensus.cc:2454] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:44.817442  7458 raft_consensus.cc:2454] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:44.817873  6866 leader_election.cc:302] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:44.818202  7937 raft_consensus.cc:2781] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:44.818322  7937 raft_consensus.cc:482] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:44.818419  7937 raft_consensus.cc:3037] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.907344  7937 raft_consensus.cc:504] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.908093  7937 leader_election.cc:288] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Requested vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083)
I20240502 08:16:44.908704  7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2c1cbd9c14fddb57a7d9c103fa496" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:44.908694  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdc2c1cbd9c14fddb57a7d9c103fa496" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:44.908923  7195 raft_consensus.cc:3037] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.908914  7458 raft_consensus.cc:3037] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:44.910845  7458 raft_consensus.cc:2454] T fdc2c1cbd9c14fddb57a7d9c103fa496 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:44.911317  6867 leader_election.cc:302] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:44.911645  7937 raft_consensus.cc:2781] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:44.912060  7937 raft_consensus.cc:686] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [term 1 LEADER]: Becoming Leader. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Running, Role: LEADER
I20240502 08:16:44.912348  7937 consensus_queue.cc:237] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:44.915158  6785 catalog_manager.cc:5484] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc93fe73601544fe8a8aa013f8de095b (127.6.151.65). New cstate: current_term: 1 leader_uuid: "cc93fe73601544fe8a8aa013f8de095b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:45.013553  7195 raft_consensus.cc:2454] T fdc2c1cbd9c14fddb57a7d9c103fa496 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:45.120703  8058 consensus_queue.cc:1035] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.127352  8058 consensus_queue.cc:1035] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.163081  8036 consensus_queue.cc:1035] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.176314  8092 consensus_queue.cc:1035] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:45.389475  8082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:45.389873  8082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:45.399341  8082 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 08:16:45.468662  8121 consensus_queue.cc:1035] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:45.489080  7937 consensus_queue.cc:1035] T fdc2c1cbd9c14fddb57a7d9c103fa496 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 304.96 ms
  time per row: 0.15248 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:16:46.849941  6783 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:46.859499  7307 tablet_service.cc:1467] Processing CreateTablet for tablet da11fa181227431f9119eba713d32e1a (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.859470  7045 tablet_service.cc:1467] Processing CreateTablet for tablet da11fa181227431f9119eba713d32e1a (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.859956  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da11fa181227431f9119eba713d32e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.859961  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da11fa181227431f9119eba713d32e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.860937  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 18202248017f434b9fbc36f64a5eb17b (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.861392  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18202248017f434b9fbc36f64a5eb17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.864990  6913 tablet_service.cc:1467] Processing CreateTablet for tablet 18202248017f434b9fbc36f64a5eb17b (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.865023  7569 tablet_service.cc:1467] Processing CreateTablet for tablet da11fa181227431f9119eba713d32e1a (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.865099  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 18202248017f434b9fbc36f64a5eb17b (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.865456  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18202248017f434b9fbc36f64a5eb17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.865453  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da11fa181227431f9119eba713d32e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.865469  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18202248017f434b9fbc36f64a5eb17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.871083  7568 tablet_service.cc:1467] Processing CreateTablet for tablet 05abbc25f6f6470590713c894affd936 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.871464  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05abbc25f6f6470590713c894affd936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.875252  7306 tablet_service.cc:1467] Processing CreateTablet for tablet 05abbc25f6f6470590713c894affd936 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.875780  7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05abbc25f6f6470590713c894affd936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.876055  8168 tablet_bootstrap.cc:492] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:46.879442  8166 tablet_bootstrap.cc:492] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:46.879431  7044 tablet_service.cc:1467] Processing CreateTablet for tablet 05abbc25f6f6470590713c894affd936 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b8254ec8d2f84312b7cb570b640854e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:46.879906  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05abbc25f6f6470590713c894affd936. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:46.881376  8168 tablet_bootstrap.cc:654] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.881145  8166 tablet_bootstrap.cc:654] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.889375  8168 tablet_bootstrap.cc:492] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:46.889549  8168 ts_tablet_manager.cc:1367] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.014s	user 0.001s	sys 0.008s
I20240502 08:16:46.890373  8168 raft_consensus.cc:348] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:46.890583  8168 raft_consensus.cc:374] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.890703  8168 raft_consensus.cc:729] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.890951  8168 consensus_queue.cc:260] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:46.891638  8168 ts_tablet_manager.cc:1398] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:46.894778  8167 tablet_bootstrap.cc:492] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:46.896543  8172 tablet_bootstrap.cc:492] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:46.896589  8167 tablet_bootstrap.cc:654] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.897621  8173 tablet_bootstrap.cc:492] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:46.901196  8173 tablet_bootstrap.cc:654] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.901389  8172 tablet_bootstrap.cc:654] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.903036  8169 tablet_bootstrap.cc:492] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:46.904727  8167 tablet_bootstrap.cc:492] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:46.905048  8167 ts_tablet_manager.cc:1367] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.026s	user 0.003s	sys 0.002s
I20240502 08:16:46.905304  8169 tablet_bootstrap.cc:654] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.907750  8166 tablet_bootstrap.cc:492] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:46.907797  8167 raft_consensus.cc:348] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:46.908038  8166 ts_tablet_manager.cc:1367] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.016s
I20240502 08:16:46.908766  8167 raft_consensus.cc:374] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.908917  8167 raft_consensus.cc:729] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.909281  8172 tablet_bootstrap.cc:492] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:46.909255  8166 raft_consensus.cc:348] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.909471  8172 ts_tablet_manager.cc:1367] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.000s
I20240502 08:16:46.909407  8167 consensus_queue.cc:260] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:46.909507  8166 raft_consensus.cc:374] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.909725  8166 raft_consensus.cc:729] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.909961  8167 ts_tablet_manager.cc:1398] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 08:16:46.910074  8166 consensus_queue.cc:260] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.911562  8166 ts_tablet_manager.cc:1398] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:16:46.911865  8166 tablet_bootstrap.cc:492] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:46.911757  8172 raft_consensus.cc:348] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.912025  8172 raft_consensus.cc:374] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.912159  8172 raft_consensus.cc:729] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.912439  8172 consensus_queue.cc:260] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.912786  8169 tablet_bootstrap.cc:492] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:46.912930  8169 ts_tablet_manager.cc:1367] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 08:16:46.913959  8172 ts_tablet_manager.cc:1398] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:16:46.914268  8166 tablet_bootstrap.cc:654] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.914278  8172 tablet_bootstrap.cc:492] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:46.915722  8169 raft_consensus.cc:348] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:46.916011  8169 raft_consensus.cc:374] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.916203  8169 raft_consensus.cc:729] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.916565  8169 consensus_queue.cc:260] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:46.918929  8169 ts_tablet_manager.cc:1398] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 08:16:46.925590  8173 tablet_bootstrap.cc:492] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:46.925768  8173 ts_tablet_manager.cc:1367] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.000s
I20240502 08:16:46.926705  8173 raft_consensus.cc:348] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.926980  8173 raft_consensus.cc:374] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.927071  8173 raft_consensus.cc:729] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.927837  8172 tablet_bootstrap.cc:654] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.928308  8166 tablet_bootstrap.cc:492] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:46.928489  8166 ts_tablet_manager.cc:1367] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.005s
I20240502 08:16:46.928982  8173 consensus_queue.cc:260] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.929215  8166 raft_consensus.cc:348] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.929432  8166 raft_consensus.cc:374] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.929526  8166 raft_consensus.cc:729] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.929639  8173 ts_tablet_manager.cc:1398] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:16:46.929762  8166 consensus_queue.cc:260] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.929908  8173 tablet_bootstrap.cc:492] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:46.930238  8166 ts_tablet_manager.cc:1398] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:16:46.931524  8173 tablet_bootstrap.cc:654] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:46.933480  8173 tablet_bootstrap.cc:492] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:46.933632  8173 ts_tablet_manager.cc:1367] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:16:46.934379  8173 raft_consensus.cc:348] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.934593  8173 raft_consensus.cc:374] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.934700  8173 raft_consensus.cc:729] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.934973  8173 consensus_queue.cc:260] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.935863  8172 tablet_bootstrap.cc:492] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:46.936062  8172 ts_tablet_manager.cc:1367] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.022s	user 0.003s	sys 0.000s
I20240502 08:16:46.936847  8172 raft_consensus.cc:348] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.937110  8172 raft_consensus.cc:374] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:46.937197  8172 raft_consensus.cc:729] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:46.937501  8172 consensus_queue.cc:260] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.937920  8172 ts_tablet_manager.cc:1398] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:46.938274  8173 ts_tablet_manager.cc:1398] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:16:46.938351  8096 raft_consensus.cc:482] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.938527  8096 raft_consensus.cc:504] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:46.939268  8096 leader_election.cc:288] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:46.939846  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18202248017f434b9fbc36f64a5eb17b" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:46.940081  6932 raft_consensus.cc:2454] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:46.940057  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18202248017f434b9fbc36f64a5eb17b" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:46.940326  7195 raft_consensus.cc:2454] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:46.940779  7391 leader_election.cc:302] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:46.941062  8096 raft_consensus.cc:2781] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.941166  8096 raft_consensus.cc:482] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.941226  8096 raft_consensus.cc:3037] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:46.942728  8087 raft_consensus.cc:482] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:46.942852  8087 raft_consensus.cc:504] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:46.943524  8087 leader_election.cc:288] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:46.943920  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05abbc25f6f6470590713c894affd936" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:46.943929  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05abbc25f6f6470590713c894affd936" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:46.944164  7064 raft_consensus.cc:2454] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 0.
I20240502 08:16:46.944175  7326 raft_consensus.cc:2454] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 0.
I20240502 08:16:46.944568  7524 leader_election.cc:302] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:46.944803  8087 raft_consensus.cc:2781] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:46.944911  8087 raft_consensus.cc:482] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:46.945016  8087 raft_consensus.cc:3037] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.029204  8087 raft_consensus.cc:504] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:47.029210  8096 raft_consensus.cc:504] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:47.029942  8087 leader_election.cc:288] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Requested vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:47.029992  8096 leader_election.cc:288] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Requested vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:47.030696  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05abbc25f6f6470590713c894affd936" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:47.030793  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18202248017f434b9fbc36f64a5eb17b" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:47.030810  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05abbc25f6f6470590713c894affd936" candidate_uuid: "881e59a197394322ade814528d5e2cdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:47.030956  7064 raft_consensus.cc:3037] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.031031  7326 raft_consensus.cc:3037] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.031030  7195 raft_consensus.cc:3037] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.033818  7326 raft_consensus.cc:2454] T 05abbc25f6f6470590713c894affd936 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:47.033811  7064 raft_consensus.cc:2454] T 05abbc25f6f6470590713c894affd936 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881e59a197394322ade814528d5e2cdf in term 1.
I20240502 08:16:47.034209  7195 raft_consensus.cc:2454] T 18202248017f434b9fbc36f64a5eb17b P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:47.034323  7524 leader_election.cc:302] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:47.034518  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18202248017f434b9fbc36f64a5eb17b" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:47.034725  7391 leader_election.cc:302] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:47.034780  8087 raft_consensus.cc:2781] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:47.034794  6932 raft_consensus.cc:3037] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.035050  8096 raft_consensus.cc:2781] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:47.035197  8096 raft_consensus.cc:686] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [term 1 LEADER]: Becoming Leader. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Running, Role: LEADER
I20240502 08:16:47.035552  8096 consensus_queue.cc:237] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:47.035694  8087 raft_consensus.cc:686] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [term 1 LEADER]: Becoming Leader. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Running, Role: LEADER
I20240502 08:16:47.035997  8087 consensus_queue.cc:237] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:47.039008  6785 catalog_manager.cc:5484] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b21a0f245140b293e607e8f554fa1f (127.6.151.69). New cstate: current_term: 1 leader_uuid: "a5b21a0f245140b293e607e8f554fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:47.039237  6783 catalog_manager.cc:5484] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 881e59a197394322ade814528d5e2cdf (127.6.151.70). New cstate: current_term: 1 leader_uuid: "881e59a197394322ade814528d5e2cdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:47.110934  8000 raft_consensus.cc:482] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:47.111153  8000 raft_consensus.cc:504] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:47.111733  8000 leader_election.cc:288] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:47.112254  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da11fa181227431f9119eba713d32e1a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:47.112232  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da11fa181227431f9119eba713d32e1a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:47.112479  7064 raft_consensus.cc:2454] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:47.112504  7589 raft_consensus.cc:2454] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:47.112927  7262 leader_election.cc:302] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:47.113193  8000 raft_consensus.cc:2781] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:47.113289  8000 raft_consensus.cc:482] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:47.113394  8000 raft_consensus.cc:3037] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.128682  6932 raft_consensus.cc:2454] T 18202248017f434b9fbc36f64a5eb17b P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:47.128682  8000 raft_consensus.cc:504] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:47.129570  8000 leader_election.cc:288] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:47.130089  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da11fa181227431f9119eba713d32e1a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:47.130053  7064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da11fa181227431f9119eba713d32e1a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:47.130280  7064 raft_consensus.cc:3037] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.130306  7589 raft_consensus.cc:3037] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:47.133067  7589 raft_consensus.cc:2454] T da11fa181227431f9119eba713d32e1a P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:47.133513  7262 leader_election.cc:302] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:47.133796  8000 raft_consensus.cc:2781] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:47.134177  8000 raft_consensus.cc:686] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:47.134471  8000 consensus_queue.cc:237] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:47.137264  6783 catalog_manager.cc:5484] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:47.233994  7064 raft_consensus.cc:2454] T da11fa181227431f9119eba713d32e1a P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:47.477198  8096 consensus_queue.cc:1035] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:47.483160  8096 consensus_queue.cc:1035] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:47.487315  8087 consensus_queue.cc:1035] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:47.494072  8183 consensus_queue.cc:1035] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:47.519600  8189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:47.519965  8189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:47.529088  8189 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 08:16:47.641844  8000 consensus_queue.cc:1035] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:47.657521  8223 consensus_queue.cc:1035] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:47.750838  7391 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.67:36791, user_credentials={real_user=slave}} blocked reactor thread for 52722.3us

INSERT report
    rows total: 2000
    time total: 288.315 ms
  time per row: 0.144157 ms
I20240502 08:16:47.973942  6783 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:47.982519  6913 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3bec7be4924971bde466a43efaa585 (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.982936  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3bec7be4924971bde466a43efaa585 (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.983048  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3bec7be4924971bde466a43efaa585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.983320  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3bec7be4924971bde466a43efaa585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.983234  7568 tablet_service.cc:1467] Processing CreateTablet for tablet 18cf97f591e04ac6bb509d9d258df83a (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.983595  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cf97f591e04ac6bb509d9d258df83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.988185  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3bec7be4924971bde466a43efaa585 (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.988641  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3bec7be4924971bde466a43efaa585. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.991091  8260 tablet_bootstrap.cc:492] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:47.993006  8260 tablet_bootstrap.cc:654] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.994398  8259 tablet_bootstrap.cc:492] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:47.995919  7306 tablet_service.cc:1467] Processing CreateTablet for tablet 18cf97f591e04ac6bb509d9d258df83a (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.996156  7044 tablet_service.cc:1467] Processing CreateTablet for tablet 18cf97f591e04ac6bb509d9d258df83a (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:47.996354  7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cf97f591e04ac6bb509d9d258df83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.996518  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cf97f591e04ac6bb509d9d258df83a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:47.997246  8263 tablet_bootstrap.cc:492] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:47.998996  8263 tablet_bootstrap.cc:654] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:47.999522  8262 tablet_bootstrap.cc:492] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:48.000653  7175 tablet_service.cc:1467] Processing CreateTablet for tablet e233e1477fbb4e1fa5c11a788c56b7e0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.001291  7437 tablet_service.cc:1467] Processing CreateTablet for tablet e233e1477fbb4e1fa5c11a788c56b7e0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.001780  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e233e1477fbb4e1fa5c11a788c56b7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.004617  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e233e1477fbb4e1fa5c11a788c56b7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.005903  6913 tablet_service.cc:1467] Processing CreateTablet for tablet e233e1477fbb4e1fa5c11a788c56b7e0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=af47fe372d2c48ae891709379803be74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:48.006508  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e233e1477fbb4e1fa5c11a788c56b7e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:48.009912  8263 tablet_bootstrap.cc:492] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:48.010118  8263 ts_tablet_manager.cc:1367] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.003s
I20240502 08:16:48.010979  8263 raft_consensus.cc:348] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.011319  8263 raft_consensus.cc:374] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.011453  8263 raft_consensus.cc:729] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.011771  8263 consensus_queue.cc:260] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.012435  8263 ts_tablet_manager.cc:1398] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:48.016535  8262 tablet_bootstrap.cc:654] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.019809  8263 tablet_bootstrap.cc:492] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:48.020799  8266 tablet_bootstrap.cc:492] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:48.021036  8259 tablet_bootstrap.cc:654] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.021840  8260 tablet_bootstrap.cc:492] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:48.021886  8263 tablet_bootstrap.cc:654] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.022107  8260 ts_tablet_manager.cc:1367] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.032s	user 0.004s	sys 0.000s
I20240502 08:16:48.023018  8266 tablet_bootstrap.cc:654] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.022986  8260 raft_consensus.cc:348] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.024755  8260 raft_consensus.cc:374] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.024940  8260 raft_consensus.cc:729] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.025285  8260 consensus_queue.cc:260] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.027369  8260 ts_tablet_manager.cc:1398] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:16:48.029172  8266 tablet_bootstrap.cc:492] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:48.029739  8266 ts_tablet_manager.cc:1367] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240502 08:16:48.031143  8259 tablet_bootstrap.cc:492] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:48.031354  8259 ts_tablet_manager.cc:1367] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.037s	user 0.000s	sys 0.004s
I20240502 08:16:48.031679  8262 tablet_bootstrap.cc:492] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:48.031874  8262 ts_tablet_manager.cc:1367] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.032s	user 0.007s	sys 0.000s
I20240502 08:16:48.032326  8259 raft_consensus.cc:348] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.032687  8262 raft_consensus.cc:348] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.032804  8259 raft_consensus.cc:374] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.032951  8262 raft_consensus.cc:374] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.032935  8259 raft_consensus.cc:729] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.033097  8262 raft_consensus.cc:729] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.033273  8263 tablet_bootstrap.cc:492] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:48.033192  8259 consensus_queue.cc:260] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.033521  8263 ts_tablet_manager.cc:1367] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.002s
I20240502 08:16:48.033399  8262 consensus_queue.cc:260] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.034265  8263 raft_consensus.cc:348] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.034508  8259 ts_tablet_manager.cc:1398] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:16:48.034632  8263 raft_consensus.cc:374] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.035140  8262 ts_tablet_manager.cc:1398] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:16:48.035447  8262 tablet_bootstrap.cc:492] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:48.035653  8265 tablet_bootstrap.cc:492] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:48.037191  8262 tablet_bootstrap.cc:654] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.037513  8265 tablet_bootstrap.cc:654] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.040060  8266 raft_consensus.cc:348] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.040356  8266 raft_consensus.cc:374] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.040490  8266 raft_consensus.cc:729] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.040745  8259 tablet_bootstrap.cc:492] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:48.040771  8266 consensus_queue.cc:260] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.041966  8233 raft_consensus.cc:482] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.042094  8233 raft_consensus.cc:504] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.042481  8259 tablet_bootstrap.cc:654] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:48.042727  8233 leader_election.cc:288] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:48.043942  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cf97f591e04ac6bb509d9d258df83a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:48.044190  7589 raft_consensus.cc:2454] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:48.044271  8263 raft_consensus.cc:729] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.044538  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cf97f591e04ac6bb509d9d258df83a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:48.044587  8263 consensus_queue.cc:260] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.045249  8263 ts_tablet_manager.cc:1398] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.000s
I20240502 08:16:48.046240  7262 leader_election.cc:302] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
W20240502 08:16:48.046587  7262 leader_election.cc:341] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:48.048451  8233 raft_consensus.cc:2781] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.048601  8233 raft_consensus.cc:482] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.048702  8233 raft_consensus.cc:3037] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.052096  8233 raft_consensus.cc:504] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.052223  8259 tablet_bootstrap.cc:492] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:48.052135  8265 tablet_bootstrap.cc:492] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:48.052376  8265 ts_tablet_manager.cc:1367] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.000s
I20240502 08:16:48.052387  8259 ts_tablet_manager.cc:1367] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.000s
I20240502 08:16:48.053236  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cf97f591e04ac6bb509d9d258df83a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:48.053205  8259 raft_consensus.cc:348] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.053426  8259 raft_consensus.cc:374] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.053279  8265 raft_consensus.cc:348] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.053459  7589 raft_consensus.cc:3037] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.053520  8265 raft_consensus.cc:374] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.053542  8259 raft_consensus.cc:729] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.053650  8265 raft_consensus.cc:729] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.053841  8259 consensus_queue.cc:260] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.053992  8265 consensus_queue.cc:260] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.054589  8265 ts_tablet_manager.cc:1398] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:48.056298  7589 raft_consensus.cc:2454] T 18cf97f591e04ac6bb509d9d258df83a P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:48.056452  8259 ts_tablet_manager.cc:1398] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:16:48.056954  7262 leader_election.cc:302] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:48.057336  8223 raft_consensus.cc:2781] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.057545  8223 raft_consensus.cc:686] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:48.057812  8223 consensus_queue.cc:237] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:48.060693  8233 leader_election.cc:288] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:48.060998  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cf97f591e04ac6bb509d9d258df83a" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:48.061199  8266 ts_tablet_manager.cc:1398] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.031s	user 0.002s	sys 0.000s
I20240502 08:16:48.061208  7065 raft_consensus.cc:3037] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.061148  6783 catalog_manager.cc:5484] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:48.063566  7065 raft_consensus.cc:2454] T 18cf97f591e04ac6bb509d9d258df83a P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:48.064464  8262 tablet_bootstrap.cc:492] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:48.064652  8262 ts_tablet_manager.cc:1367] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.029s	user 0.004s	sys 0.000s
I20240502 08:16:48.065562  8262 raft_consensus.cc:348] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.065793  8262 raft_consensus.cc:374] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:48.065943  8262 raft_consensus.cc:729] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:48.066228  8262 consensus_queue.cc:260] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.066857  8262 ts_tablet_manager.cc:1398] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 08:16:48.070891  7960 raft_consensus.cc:482] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.071055  7960 raft_consensus.cc:504] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.071660  7960 leader_election.cc:288] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:48.072172  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e233e1477fbb4e1fa5c11a788c56b7e0" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:48.072434  6932 raft_consensus.cc:2454] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:48.072875  7129 leader_election.cc:302] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:48.072875  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e233e1477fbb4e1fa5c11a788c56b7e0" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:48.073124  7457 raft_consensus.cc:2454] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:48.073163  7960 raft_consensus.cc:2781] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.073261  7960 raft_consensus.cc:482] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.073351  7960 raft_consensus.cc:3037] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.075315  7960 raft_consensus.cc:504] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.075791  7960 leader_election.cc:288] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:48.076206  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e233e1477fbb4e1fa5c11a788c56b7e0" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:48.076228  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e233e1477fbb4e1fa5c11a788c56b7e0" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:48.076377  7457 raft_consensus.cc:3037] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.076406  6933 raft_consensus.cc:3037] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.078155  6933 raft_consensus.cc:2454] T e233e1477fbb4e1fa5c11a788c56b7e0 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:48.078171  7457 raft_consensus.cc:2454] T e233e1477fbb4e1fa5c11a788c56b7e0 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:48.078526  7130 leader_election.cc:302] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:48.078847  7960 raft_consensus.cc:2781] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.079270  7960 raft_consensus.cc:686] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [term 1 LEADER]: Becoming Leader. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Running, Role: LEADER
I20240502 08:16:48.079557  7960 consensus_queue.cc:237] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:48.082175  6783 catalog_manager.cc:5484] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 136680942cc947b0b19be2f50bb114eb (127.6.151.67). New cstate: current_term: 1 leader_uuid: "136680942cc947b0b19be2f50bb114eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:48.210436  8235 raft_consensus.cc:482] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:48.210669  8235 raft_consensus.cc:504] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.211431  8235 leader_election.cc:288] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:48.212092  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3bec7be4924971bde466a43efaa585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:48.212092  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3bec7be4924971bde466a43efaa585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:48.212340  7195 raft_consensus.cc:2454] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:48.212340  6933 raft_consensus.cc:2454] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:48.212760  7391 leader_election.cc:302] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:48.212998  8235 raft_consensus.cc:2781] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:48.213095  8235 raft_consensus.cc:482] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:48.213176  8235 raft_consensus.cc:3037] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.215152  8235 raft_consensus.cc:504] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.215667  8235 leader_election.cc:288] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Requested vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:48.216131  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3bec7be4924971bde466a43efaa585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:48.216207  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3bec7be4924971bde466a43efaa585" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:48.216368  6933 raft_consensus.cc:3037] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.216382  7195 raft_consensus.cc:3037] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:48.219235  7195 raft_consensus.cc:2454] T 1b3bec7be4924971bde466a43efaa585 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:48.219233  6933 raft_consensus.cc:2454] T 1b3bec7be4924971bde466a43efaa585 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:48.219640  7391 leader_election.cc:302] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:48.219888  8235 raft_consensus.cc:2781] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:48.220044  8235 raft_consensus.cc:686] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [term 1 LEADER]: Becoming Leader. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Running, Role: LEADER
I20240502 08:16:48.220307  8235 consensus_queue.cc:237] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:48.223335  6783 catalog_manager.cc:5484] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b21a0f245140b293e607e8f554fa1f (127.6.151.69). New cstate: current_term: 1 leader_uuid: "a5b21a0f245140b293e607e8f554fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:48.437891  8223 consensus_queue.cc:1035] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.442420  8238 consensus_queue.cc:1035] T 18cf97f591e04ac6bb509d9d258df83a P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.542557  7960 consensus_queue.cc:1035] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.548931  7960 consensus_queue.cc:1035] T e233e1477fbb4e1fa5c11a788c56b7e0 P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.608255  8235 consensus_queue.cc:1035] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:48.612931  8235 consensus_queue.cc:1035] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:48.647603  8278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:48.647998  8278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:48.655949  8278 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'

INSERT report
    rows total: 2000
    time total: 235.37 ms
  time per row: 0.117685 ms
I20240502 08:16:49.043089  6783 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:49.052304  7044 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7648c5b0c442f2aa908e60d584addf (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.052516  7568 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7648c5b0c442f2aa908e60d584addf (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.052824  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7648c5b0c442f2aa908e60d584addf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.052945  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7648c5b0c442f2aa908e60d584addf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.053169  7306 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7648c5b0c442f2aa908e60d584addf (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.053601  7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7648c5b0c442f2aa908e60d584addf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.053860  6913 tablet_service.cc:1467] Processing CreateTablet for tablet 55bcef87afbf469384cc6e067145fbdc (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.054261  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bcef87afbf469384cc6e067145fbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.055097  7307 tablet_service.cc:1467] Processing CreateTablet for tablet d203d388047241268160248093e177a7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.055498  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d203d388047241268160248093e177a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.058432  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 55bcef87afbf469384cc6e067145fbdc (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.058910  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bcef87afbf469384cc6e067145fbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.059598  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 55bcef87afbf469384cc6e067145fbdc (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.060151  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bcef87afbf469384cc6e067145fbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.060056  7569 tablet_service.cc:1467] Processing CreateTablet for tablet d203d388047241268160248093e177a7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.060451  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d203d388047241268160248093e177a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.064896  7045 tablet_service.cc:1467] Processing CreateTablet for tablet d203d388047241268160248093e177a7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=eef055c3dc664003b0b7c109a9a96053]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:49.065338  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d203d388047241268160248093e177a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:49.073171  8337 tablet_bootstrap.cc:492] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:49.075842  8337 tablet_bootstrap.cc:654] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.079525  8341 tablet_bootstrap.cc:492] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:49.082901  8341 tablet_bootstrap.cc:654] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.083667  8339 tablet_bootstrap.cc:492] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:49.084481  8342 tablet_bootstrap.cc:492] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:49.085426  8338 tablet_bootstrap.cc:492] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:49.087016  8340 tablet_bootstrap.cc:492] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:49.087810  8338 tablet_bootstrap.cc:654] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.088078  8339 tablet_bootstrap.cc:654] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.088403  8342 tablet_bootstrap.cc:654] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.090844  8338 tablet_bootstrap.cc:492] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:49.091065  8338 ts_tablet_manager.cc:1367] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 08:16:49.091972  8338 raft_consensus.cc:348] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.092166  8340 tablet_bootstrap.cc:654] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.092288  8338 raft_consensus.cc:374] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.092496  8338 raft_consensus.cc:729] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.092768  8338 consensus_queue.cc:260] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.092933  8337 tablet_bootstrap.cc:492] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:49.093117  8337 ts_tablet_manager.cc:1367] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.004s
I20240502 08:16:49.093529  8338 ts_tablet_manager.cc:1398] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:49.093925  8337 raft_consensus.cc:348] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.094189  8337 raft_consensus.cc:374] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.094314  8337 raft_consensus.cc:729] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.094588  8337 consensus_queue.cc:260] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.095439  8337 ts_tablet_manager.cc:1398] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 08:16:49.095708  8337 tablet_bootstrap.cc:492] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:49.096480  8339 tablet_bootstrap.cc:492] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:49.096722  8339 ts_tablet_manager.cc:1367] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.013s	user 0.000s	sys 0.007s
I20240502 08:16:49.097883  8340 tablet_bootstrap.cc:492] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:49.098014  8337 tablet_bootstrap.cc:654] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.098044  8340 ts_tablet_manager.cc:1367] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.002s
I20240502 08:16:49.098872  8340 raft_consensus.cc:348] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.099092  8340 raft_consensus.cc:374] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.099210  8340 raft_consensus.cc:729] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.099468  8340 consensus_queue.cc:260] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.100061  8340 ts_tablet_manager.cc:1398] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:49.100958  8341 tablet_bootstrap.cc:492] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:49.101167  8341 ts_tablet_manager.cc:1367] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.022s	user 0.002s	sys 0.005s
I20240502 08:16:49.102735  8342 tablet_bootstrap.cc:492] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:49.102948  8342 ts_tablet_manager.cc:1367] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.000s
I20240502 08:16:49.103741  8342 raft_consensus.cc:348] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.104506  8341 raft_consensus.cc:348] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.104799  8341 raft_consensus.cc:374] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.104923  8341 raft_consensus.cc:729] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.105204  8341 consensus_queue.cc:260] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.105563  8339 raft_consensus.cc:348] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.105854  8342 raft_consensus.cc:374] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.106022  8342 raft_consensus.cc:729] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.106307  8341 ts_tablet_manager.cc:1398] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 08:16:49.106369  8342 consensus_queue.cc:260] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.106521  8339 raft_consensus.cc:374] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.106693  8339 raft_consensus.cc:729] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.107358  8337 tablet_bootstrap.cc:492] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:49.107538  8337 ts_tablet_manager.cc:1367] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.008s
I20240502 08:16:49.107563  8342 ts_tablet_manager.cc:1398] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:16:49.107964  8342 tablet_bootstrap.cc:492] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:49.108342  8337 raft_consensus.cc:348] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.109375  8337 raft_consensus.cc:374] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.109469  8337 raft_consensus.cc:729] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.107018  8339 consensus_queue.cc:260] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.111308  8342 tablet_bootstrap.cc:654] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.111752  8337 consensus_queue.cc:260] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.113499  8337 ts_tablet_manager.cc:1398] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.002s
I20240502 08:16:49.114529  8339 ts_tablet_manager.cc:1398] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.016s	user 0.002s	sys 0.000s
I20240502 08:16:49.114917  8339 tablet_bootstrap.cc:492] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:49.115949  8342 tablet_bootstrap.cc:492] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:49.116118  8342 ts_tablet_manager.cc:1367] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.008s	user 0.005s	sys 0.000s
I20240502 08:16:49.116670  8342 raft_consensus.cc:348] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.116876  8342 raft_consensus.cc:374] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.117554  8339 tablet_bootstrap.cc:654] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:49.118036  8342 raft_consensus.cc:729] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.118333  8342 consensus_queue.cc:260] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.118867  8342 ts_tablet_manager.cc:1398] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:49.119726  8339 tablet_bootstrap.cc:492] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:49.119881  8339 ts_tablet_manager.cc:1367] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:16:49.120512  8339 raft_consensus.cc:348] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.120723  8339 raft_consensus.cc:374] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:49.120801  8339 raft_consensus.cc:729] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:49.121039  8339 consensus_queue.cc:260] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.121563  8339 ts_tablet_manager.cc:1398] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:16:49.172870  8121 raft_consensus.cc:482] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.173190  8121 raft_consensus.cc:504] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.174042  8121 leader_election.cc:288] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:49.174698  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bcef87afbf469384cc6e067145fbdc" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:49.174608  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bcef87afbf469384cc6e067145fbdc" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:49.174957  7457 raft_consensus.cc:2454] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:49.175068  7195 raft_consensus.cc:2454] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:49.175448  6867 leader_election.cc:302] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:49.175768  8121 raft_consensus.cc:2781] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.175870  8121 raft_consensus.cc:482] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.176010  8121 raft_consensus.cc:3037] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.177963  8121 raft_consensus.cc:504] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.178529  8121 leader_election.cc:288] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:49.179092  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bcef87afbf469384cc6e067145fbdc" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:49.179090  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55bcef87afbf469384cc6e067145fbdc" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:49.179317  7195 raft_consensus.cc:3037] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.179317  7457 raft_consensus.cc:3037] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.182097  7457 raft_consensus.cc:2454] T 55bcef87afbf469384cc6e067145fbdc P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:49.182097  7195 raft_consensus.cc:2454] T 55bcef87afbf469384cc6e067145fbdc P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:49.182562  6867 leader_election.cc:302] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:49.182900  8121 raft_consensus.cc:2781] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:49.183349  8121 raft_consensus.cc:686] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [term 1 LEADER]: Becoming Leader. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Running, Role: LEADER
I20240502 08:16:49.183926  8121 consensus_queue.cc:237] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:49.187081  6789 catalog_manager.cc:5484] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc93fe73601544fe8a8aa013f8de095b (127.6.151.65). New cstate: current_term: 1 leader_uuid: "cc93fe73601544fe8a8aa013f8de095b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:49.480897  8326 raft_consensus.cc:482] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.481201  8326 raft_consensus.cc:504] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.481868  8326 leader_election.cc:288] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:49.482318  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7648c5b0c442f2aa908e60d584addf" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:49.482355  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7648c5b0c442f2aa908e60d584addf" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:49.482542  7065 raft_consensus.cc:2454] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:49.482595  7589 raft_consensus.cc:2454] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:49.482960  7262 leader_election.cc:302] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:49.483228  8326 raft_consensus.cc:2781] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.483330  8326 raft_consensus.cc:482] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.483413  8326 raft_consensus.cc:3037] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.485200  8326 raft_consensus.cc:504] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.485700  8326 leader_election.cc:288] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:49.486254  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7648c5b0c442f2aa908e60d584addf" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:49.486271  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d7648c5b0c442f2aa908e60d584addf" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:49.486469  7065 raft_consensus.cc:3037] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.486485  7589 raft_consensus.cc:3037] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.489257  7589 raft_consensus.cc:2454] T 0d7648c5b0c442f2aa908e60d584addf P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:49.489275  7065 raft_consensus.cc:2454] T 0d7648c5b0c442f2aa908e60d584addf P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:49.489688  7262 leader_election.cc:302] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:49.489959  8326 raft_consensus.cc:2781] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:49.490346  8326 raft_consensus.cc:686] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:49.490665  8326 consensus_queue.cc:237] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:49.493414  6789 catalog_manager.cc:5484] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:49.594008  8083 raft_consensus.cc:482] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:49.594233  8083 raft_consensus.cc:504] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.595000  8083 leader_election.cc:288] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:49.595428  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203d388047241268160248093e177a7" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:49.595564  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203d388047241268160248093e177a7" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:49.595669  7589 raft_consensus.cc:2454] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:49.595768  7326 raft_consensus.cc:2454] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:49.596028  7000 leader_election.cc:302] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:49.596287  8083 raft_consensus.cc:2781] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:49.596395  8083 raft_consensus.cc:482] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:49.596482  8083 raft_consensus.cc:3037] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.598675  8083 raft_consensus.cc:504] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.599227  8083 leader_election.cc:288] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Requested vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:49.599660  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203d388047241268160248093e177a7" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:49.599710  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d203d388047241268160248093e177a7" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:49.599895  7326 raft_consensus.cc:3037] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.599889  7589 raft_consensus.cc:3037] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:49.602702  7589 raft_consensus.cc:2454] T d203d388047241268160248093e177a7 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:49.602701  7326 raft_consensus.cc:2454] T d203d388047241268160248093e177a7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:49.603127  7001 leader_election.cc:302] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:49.603435  8083 raft_consensus.cc:2781] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:49.603837  8083 raft_consensus.cc:686] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [term 1 LEADER]: Becoming Leader. State: Replica: b8432324fa494be3a41163adc25b0865, State: Running, Role: LEADER
I20240502 08:16:49.604117  8083 consensus_queue.cc:237] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:49.607254  6789 catalog_manager.cc:5484] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8432324fa494be3a41163adc25b0865 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "b8432324fa494be3a41163adc25b0865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:49.618794  8121 consensus_queue.cc:1035] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:49.620374  8121 consensus_queue.cc:1035] T 55bcef87afbf469384cc6e067145fbdc P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:49.914579  8326 consensus_queue.cc:1035] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:49.922792  8326 consensus_queue.cc:1035] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:49.986040  8364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:49.986421  8364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:49.995344  8364 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 08:16:50.077301  8083 consensus_queue.cc:1035] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.084558  8356 consensus_queue.cc:1035] T d203d388047241268160248093e177a7 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 249.607 ms
  time per row: 0.124804 ms
I20240502 08:16:50.413405  6786 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:50.422009  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 7c88e7a1a04849d585c5ca9baaed7686 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.422506  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c88e7a1a04849d585c5ca9baaed7686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.422461  6913 tablet_service.cc:1467] Processing CreateTablet for tablet 7c88e7a1a04849d585c5ca9baaed7686 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.422894  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c88e7a1a04849d585c5ca9baaed7686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.423820  7568 tablet_service.cc:1467] Processing CreateTablet for tablet 834e8b656bcb417184b50efbead45f86 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.424139  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834e8b656bcb417184b50efbead45f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.427327  7438 tablet_service.cc:1467] Processing CreateTablet for tablet 174252fc5fcd4630b409727636aeb2f6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.427376  7044 tablet_service.cc:1467] Processing CreateTablet for tablet 834e8b656bcb417184b50efbead45f86 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.427750  7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174252fc5fcd4630b409727636aeb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.427656  7306 tablet_service.cc:1467] Processing CreateTablet for tablet 834e8b656bcb417184b50efbead45f86 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.427805  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834e8b656bcb417184b50efbead45f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.428051  7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 834e8b656bcb417184b50efbead45f86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.428066  6912 tablet_service.cc:1467] Processing CreateTablet for tablet 174252fc5fcd4630b409727636aeb2f6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.428418  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174252fc5fcd4630b409727636aeb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.434988  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 7c88e7a1a04849d585c5ca9baaed7686 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.435411  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c88e7a1a04849d585c5ca9baaed7686. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.438973  7176 tablet_service.cc:1467] Processing CreateTablet for tablet 174252fc5fcd4630b409727636aeb2f6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=300ac22fc1404649943f1fe2ff9b04c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:50.439507  7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 174252fc5fcd4630b409727636aeb2f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:50.441779  8432 tablet_bootstrap.cc:492] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:50.444360  8432 tablet_bootstrap.cc:654] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.452984  8432 tablet_bootstrap.cc:492] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:50.453298  8432 ts_tablet_manager.cc:1367] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.008s
I20240502 08:16:50.454324  8436 tablet_bootstrap.cc:492] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:50.454320  8432 raft_consensus.cc:348] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.454682  8432 raft_consensus.cc:374] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.454842  8432 raft_consensus.cc:729] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.455183  8432 consensus_queue.cc:260] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.455847  8432 ts_tablet_manager.cc:1398] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:50.456166  8432 tablet_bootstrap.cc:492] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:50.457772  8432 tablet_bootstrap.cc:654] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.459583  8434 tablet_bootstrap.cc:492] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:50.459964  8431 tablet_bootstrap.cc:492] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:50.461705  8431 tablet_bootstrap.cc:654] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.463228  8433 tablet_bootstrap.cc:492] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:50.463837  8436 tablet_bootstrap.cc:654] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.464365  8434 tablet_bootstrap.cc:654] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.465255  8430 tablet_bootstrap.cc:492] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:50.465482  8433 tablet_bootstrap.cc:654] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.467805  8430 tablet_bootstrap.cc:654] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.469183  8431 tablet_bootstrap.cc:492] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:50.469759  8431 ts_tablet_manager.cc:1367] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240502 08:16:50.470912  8431 raft_consensus.cc:348] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.472941  8431 raft_consensus.cc:374] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.473261  8431 raft_consensus.cc:729] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.473729  8431 consensus_queue.cc:260] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.476238  8431 ts_tablet_manager.cc:1398] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.000s
I20240502 08:16:50.477792  8436 tablet_bootstrap.cc:492] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:50.477989  8436 ts_tablet_manager.cc:1367] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.024s	user 0.006s	sys 0.000s
I20240502 08:16:50.478796  8436 raft_consensus.cc:348] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.479300  8431 tablet_bootstrap.cc:492] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:50.479444  8436 raft_consensus.cc:374] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.479594  8436 raft_consensus.cc:729] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.479889  8436 consensus_queue.cc:260] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.487462  8430 tablet_bootstrap.cc:492] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:50.487774  8430 ts_tablet_manager.cc:1367] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.023s	user 0.000s	sys 0.008s
I20240502 08:16:50.488778  8430 raft_consensus.cc:348] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.489101  8430 raft_consensus.cc:374] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.489261  8430 raft_consensus.cc:729] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.489379  8432 tablet_bootstrap.cc:492] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:50.489545  8432 ts_tablet_manager.cc:1367] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.033s	user 0.005s	sys 0.025s
I20240502 08:16:50.489686  8434 tablet_bootstrap.cc:492] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:50.489687  8430 consensus_queue.cc:260] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.489871  8434 ts_tablet_manager.cc:1367] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.030s	user 0.004s	sys 0.004s
I20240502 08:16:50.490321  8430 ts_tablet_manager.cc:1398] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:16:50.490296  8432 raft_consensus.cc:348] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.490491  8432 raft_consensus.cc:374] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.490568  8432 raft_consensus.cc:729] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.490873  8432 consensus_queue.cc:260] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.491349  8436 ts_tablet_manager.cc:1398] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.000s
I20240502 08:16:50.491497  8434 raft_consensus.cc:348] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.491675  8436 tablet_bootstrap.cc:492] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:50.491719  8434 raft_consensus.cc:374] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.491850  8434 raft_consensus.cc:729] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.492079  8434 consensus_queue.cc:260] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.492765  8434 ts_tablet_manager.cc:1398] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:16:50.493259  8431 tablet_bootstrap.cc:654] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.494521  8432 ts_tablet_manager.cc:1398] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 08:16:50.495400  8291 raft_consensus.cc:482] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.495546  8291 raft_consensus.cc:504] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.496238  8291 leader_election.cc:288] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:50.496882  8436 tablet_bootstrap.cc:654] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:50.496912  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c88e7a1a04849d585c5ca9baaed7686" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:50.497653  8433 tablet_bootstrap.cc:492] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:50.497805  8433 ts_tablet_manager.cc:1367] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.035s	user 0.005s	sys 0.000s
I20240502 08:16:50.497740  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c88e7a1a04849d585c5ca9baaed7686" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
W20240502 08:16:50.499258  7391 leader_election.cc:341] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Illegal state: must be running to vote when last-logged opid is not known
W20240502 08:16:50.499608  7391 leader_election.cc:341] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:16:50.499732  7391 leader_election.cc:302] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f; no voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b
I20240502 08:16:50.499693  8433 raft_consensus.cc:348] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.500028  8433 raft_consensus.cc:374] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.500121  8433 raft_consensus.cc:729] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.500195  8291 raft_consensus.cc:2726] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:16:50.500360  8433 consensus_queue.cc:260] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.501196  8433 ts_tablet_manager.cc:1398] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:16:50.508971  8431 tablet_bootstrap.cc:492] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:50.509289  8431 ts_tablet_manager.cc:1367] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.030s	user 0.004s	sys 0.005s
I20240502 08:16:50.510138  8431 raft_consensus.cc:348] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.510236  8436 tablet_bootstrap.cc:492] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:50.510321  8431 raft_consensus.cc:374] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.510401  8436 ts_tablet_manager.cc:1367] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.019s	user 0.002s	sys 0.011s
I20240502 08:16:50.510397  8431 raft_consensus.cc:729] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.510669  8431 consensus_queue.cc:260] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.511180  8431 ts_tablet_manager.cc:1398] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:16:50.511147  8436 raft_consensus.cc:348] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.511363  8436 raft_consensus.cc:374] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:50.511442  8436 raft_consensus.cc:729] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:50.511665  8436 consensus_queue.cc:260] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.512236  8436 ts_tablet_manager.cc:1398] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:16:50.525954  8121 raft_consensus.cc:482] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.526124  8121 raft_consensus.cc:504] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.526798  8121 leader_election.cc:288] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:50.527220  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c88e7a1a04849d585c5ca9baaed7686" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:50.527266  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c88e7a1a04849d585c5ca9baaed7686" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:50.527421  7457 raft_consensus.cc:2454] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:50.527490  7195 raft_consensus.cc:2454] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:50.527838  6867 leader_election.cc:302] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:50.528139  8121 raft_consensus.cc:2781] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.528246  8121 raft_consensus.cc:482] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.528331  8121 raft_consensus.cc:3037] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.530184  8121 raft_consensus.cc:504] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.530817  8121 leader_election.cc:288] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:50.531255  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c88e7a1a04849d585c5ca9baaed7686" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:50.531307  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c88e7a1a04849d585c5ca9baaed7686" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:50.531448  7457 raft_consensus.cc:3037] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.531491  7195 raft_consensus.cc:3037] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.533366  7457 raft_consensus.cc:2454] T 7c88e7a1a04849d585c5ca9baaed7686 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:50.533385  7195 raft_consensus.cc:2454] T 7c88e7a1a04849d585c5ca9baaed7686 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:50.533775  6867 leader_election.cc:302] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:50.534080  8121 raft_consensus.cc:2781] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.534204  8121 raft_consensus.cc:686] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [term 1 LEADER]: Becoming Leader. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Running, Role: LEADER
I20240502 08:16:50.534471  8121 consensus_queue.cc:237] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:50.537264  6783 catalog_manager.cc:5484] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc93fe73601544fe8a8aa013f8de095b (127.6.151.65). New cstate: current_term: 1 leader_uuid: "cc93fe73601544fe8a8aa013f8de095b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:50.587999  8418 raft_consensus.cc:482] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.588184  8418 raft_consensus.cc:504] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.588810  8418 leader_election.cc:288] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:50.589316  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834e8b656bcb417184b50efbead45f86" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:50.589322  7588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834e8b656bcb417184b50efbead45f86" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:50.589532  7588 raft_consensus.cc:2454] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:50.589546  7326 raft_consensus.cc:2454] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:50.589964  7000 leader_election.cc:302] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:50.590231  8418 raft_consensus.cc:2781] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.590342  8418 raft_consensus.cc:482] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.590440  8418 raft_consensus.cc:3037] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.592406  8418 raft_consensus.cc:504] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.592942  8418 leader_election.cc:288] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Requested vote from peers 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:50.593482  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834e8b656bcb417184b50efbead45f86" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:50.593484  7588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "834e8b656bcb417184b50efbead45f86" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:50.593755  7326 raft_consensus.cc:3037] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.593755  7588 raft_consensus.cc:3037] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.596364  8291 raft_consensus.cc:482] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:50.596592  7588 raft_consensus.cc:2454] T 834e8b656bcb417184b50efbead45f86 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:50.596581  7326 raft_consensus.cc:2454] T 834e8b656bcb417184b50efbead45f86 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:50.596526  8291 raft_consensus.cc:504] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.597028  7000 leader_election.cc:302] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:50.597249  8291 leader_election.cc:288] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:50.597298  8418 raft_consensus.cc:2781] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.597452  8418 raft_consensus.cc:686] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [term 1 LEADER]: Becoming Leader. State: Replica: b8432324fa494be3a41163adc25b0865, State: Running, Role: LEADER
I20240502 08:16:50.597733  8418 consensus_queue.cc:237] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:50.598244  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174252fc5fcd4630b409727636aeb2f6" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:50.598317  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174252fc5fcd4630b409727636aeb2f6" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:50.598475  7195 raft_consensus.cc:2454] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:50.598536  6933 raft_consensus.cc:2454] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:50.598995  7391 leader_election.cc:302] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:50.599339  8291 raft_consensus.cc:2781] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:50.599493  8291 raft_consensus.cc:482] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:50.599583  8291 raft_consensus.cc:3037] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.601977  8291 raft_consensus.cc:504] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.601910  6783 catalog_manager.cc:5484] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8432324fa494be3a41163adc25b0865 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "b8432324fa494be3a41163adc25b0865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:50.602661  8291 leader_election.cc:288] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Requested vote from peers 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:50.603196  6933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174252fc5fcd4630b409727636aeb2f6" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:50.603214  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "174252fc5fcd4630b409727636aeb2f6" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:50.603394  6933 raft_consensus.cc:3037] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.603410  7195 raft_consensus.cc:3037] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:50.606168  7195 raft_consensus.cc:2454] T 174252fc5fcd4630b409727636aeb2f6 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:50.606525  7391 leader_election.cc:302] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:50.606813  8291 raft_consensus.cc:2781] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:50.607172  8291 raft_consensus.cc:686] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [term 1 LEADER]: Becoming Leader. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Running, Role: LEADER
I20240502 08:16:50.607437  8291 consensus_queue.cc:237] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:50.610353  6783 catalog_manager.cc:5484] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b21a0f245140b293e607e8f554fa1f (127.6.151.69). New cstate: current_term: 1 leader_uuid: "a5b21a0f245140b293e607e8f554fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: HEALTHY } } }
I20240502 08:16:50.624606  6933 raft_consensus.cc:2454] T 174252fc5fcd4630b409727636aeb2f6 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:50.943676  8121 consensus_queue.cc:1035] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:50.950803  8453 consensus_queue.cc:1035] T 7c88e7a1a04849d585c5ca9baaed7686 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:50.987496  8449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:50.987852  8449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:50.995625  8449 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:51.024255  8418 consensus_queue.cc:1035] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:51.032004  8291 consensus_queue.cc:1035] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 08:16:51.057337  8421 consensus_queue.cc:1035] T 834e8b656bcb417184b50efbead45f86 P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:51.065936  8447 consensus_queue.cc:1035] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 254.651 ms
  time per row: 0.127325 ms
I20240502 08:16:51.455861  6783 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:51.466578  7306 tablet_service.cc:1467] Processing CreateTablet for tablet acf47308a2aa468f9aab6bbbf66c2a22 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.467079  7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf47308a2aa468f9aab6bbbf66c2a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.467010  7307 tablet_service.cc:1467] Processing CreateTablet for tablet 723c99fd647142049bee8313ac4578a1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.467335  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723c99fd647142049bee8313ac4578a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.467793  7044 tablet_service.cc:1467] Processing CreateTablet for tablet acf47308a2aa468f9aab6bbbf66c2a22 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.468200  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf47308a2aa468f9aab6bbbf66c2a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.468098  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 53343926b3f14cfa9c3beb3de20eb35b (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.468490  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343926b3f14cfa9c3beb3de20eb35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.468545  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 53343926b3f14cfa9c3beb3de20eb35b (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.468955  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343926b3f14cfa9c3beb3de20eb35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.471097  7045 tablet_service.cc:1467] Processing CreateTablet for tablet 723c99fd647142049bee8313ac4578a1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.471498  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723c99fd647142049bee8313ac4578a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.472723  7569 tablet_service.cc:1467] Processing CreateTablet for tablet 723c99fd647142049bee8313ac4578a1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.473124  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 723c99fd647142049bee8313ac4578a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.474920  7568 tablet_service.cc:1467] Processing CreateTablet for tablet acf47308a2aa468f9aab6bbbf66c2a22 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.477363  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf47308a2aa468f9aab6bbbf66c2a22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.484347  6912 tablet_service.cc:1467] Processing CreateTablet for tablet 53343926b3f14cfa9c3beb3de20eb35b (DEFAULT_TABLE table=rebalance_test_table_9 [id=1f895d6c1229413580f9fa1948f39644]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:51.484783  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53343926b3f14cfa9c3beb3de20eb35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:51.495647  8514 tablet_bootstrap.cc:492] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:51.497097  8511 tablet_bootstrap.cc:492] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:51.497895  8514 tablet_bootstrap.cc:654] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.499001  8515 tablet_bootstrap.cc:492] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:51.500865  8515 tablet_bootstrap.cc:654] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.501946  8511 tablet_bootstrap.cc:654] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.502853  8513 tablet_bootstrap.cc:492] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:51.506927  8510 tablet_bootstrap.cc:492] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:51.507846  8512 tablet_bootstrap.cc:492] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:51.509202  8510 tablet_bootstrap.cc:654] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.513186  8511 tablet_bootstrap.cc:492] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:51.513407  8511 ts_tablet_manager.cc:1367] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.007s
I20240502 08:16:51.514258  8513 tablet_bootstrap.cc:654] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.515414  8512 tablet_bootstrap.cc:654] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.517675  8515 tablet_bootstrap.cc:492] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:51.517856  8510 tablet_bootstrap.cc:492] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:51.517872  8515 ts_tablet_manager.cc:1367] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.019s	user 0.002s	sys 0.013s
I20240502 08:16:51.518028  8510 ts_tablet_manager.cc:1367] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.011s	user 0.004s	sys 0.000s
I20240502 08:16:51.518707  8515 raft_consensus.cc:348] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.518810  8510 raft_consensus.cc:348] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.519024  8510 raft_consensus.cc:374] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.519102  8515 raft_consensus.cc:374] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.519205  8515 raft_consensus.cc:729] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.519280  8510 raft_consensus.cc:729] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.519497  8515 consensus_queue.cc:260] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.519563  8510 consensus_queue.cc:260] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.520910  8515 ts_tablet_manager.cc:1398] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:16:51.520948  8510 ts_tablet_manager.cc:1398] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:16:51.521214  8510 tablet_bootstrap.cc:492] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:51.522279  8514 tablet_bootstrap.cc:492] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:51.522446  8514 ts_tablet_manager.cc:1367] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.000s
I20240502 08:16:51.522683  8510 tablet_bootstrap.cc:654] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.522778  8511 raft_consensus.cc:348] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.523118  8511 raft_consensus.cc:374] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.523474  8511 raft_consensus.cc:729] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.524336  8511 consensus_queue.cc:260] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.525358  8511 ts_tablet_manager.cc:1398] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.012s	user 0.001s	sys 0.001s
I20240502 08:16:51.525473  8514 raft_consensus.cc:348] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.525758  8514 raft_consensus.cc:374] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.526341  8514 raft_consensus.cc:729] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.526671  8514 consensus_queue.cc:260] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.527361  8514 ts_tablet_manager.cc:1398] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.002s
I20240502 08:16:51.528323  8514 tablet_bootstrap.cc:492] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:51.529966  8513 tablet_bootstrap.cc:492] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:51.530205  8513 ts_tablet_manager.cc:1367] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.027s	user 0.002s	sys 0.004s
I20240502 08:16:51.530836  8514 tablet_bootstrap.cc:654] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.531749  8510 tablet_bootstrap.cc:492] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:51.531975  8510 ts_tablet_manager.cc:1367] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.011s	user 0.001s	sys 0.002s
I20240502 08:16:51.532794  8510 raft_consensus.cc:348] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.533071  8510 raft_consensus.cc:374] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.533185  8512 tablet_bootstrap.cc:492] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:51.533196  8510 raft_consensus.cc:729] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.533475  8510 consensus_queue.cc:260] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.534925  8510 ts_tablet_manager.cc:1398] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:51.534920  8513 raft_consensus.cc:348] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.535214  8513 raft_consensus.cc:374] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.535331  8513 raft_consensus.cc:729] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.535614  8513 consensus_queue.cc:260] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.536484  8512 ts_tablet_manager.cc:1367] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.029s	user 0.002s	sys 0.004s
I20240502 08:16:51.537449  8513 ts_tablet_manager.cc:1398] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 08:16:51.538091  8512 raft_consensus.cc:348] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.538349  8512 raft_consensus.cc:374] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.538479  8512 raft_consensus.cc:729] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.539028  8514 tablet_bootstrap.cc:492] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:51.538961  8512 consensus_queue.cc:260] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.539207  8514 ts_tablet_manager.cc:1367] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.004s
I20240502 08:16:51.539790  8512 ts_tablet_manager.cc:1398] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:16:51.540048  8512 tablet_bootstrap.cc:492] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:51.540130  8514 raft_consensus.cc:348] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.540378  8514 raft_consensus.cc:374] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.540530  8514 raft_consensus.cc:729] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.540910  8514 consensus_queue.cc:260] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.541638  8512 tablet_bootstrap.cc:654] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:51.542325  8514 ts_tablet_manager.cc:1398] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:51.559239  8512 tablet_bootstrap.cc:492] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:51.559451  8512 ts_tablet_manager.cc:1367] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.019s	user 0.000s	sys 0.003s
I20240502 08:16:51.560235  8512 raft_consensus.cc:348] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.560467  8512 raft_consensus.cc:374] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:51.560546  8512 raft_consensus.cc:729] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:51.560791  8512 consensus_queue.cc:260] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.561296  8512 ts_tablet_manager.cc:1398] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:16:51.567380  8354 raft_consensus.cc:482] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.567584  8354 raft_consensus.cc:504] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.568267  8354 leader_election.cc:288] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:51.568661  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723c99fd647142049bee8313ac4578a1" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:51.568786  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723c99fd647142049bee8313ac4578a1" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:51.568902  7065 raft_consensus.cc:2454] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:51.569051  7589 raft_consensus.cc:2454] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:51.569326  7262 leader_election.cc:302] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:51.569602  8354 raft_consensus.cc:2781] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:51.569725  8354 raft_consensus.cc:482] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:51.569799  8354 raft_consensus.cc:3037] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.572546  8354 raft_consensus.cc:504] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.573117  8354 leader_election.cc:288] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:51.573536  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723c99fd647142049bee8313ac4578a1" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:51.573611  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "723c99fd647142049bee8313ac4578a1" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:51.573761  7065 raft_consensus.cc:3037] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.573791  7589 raft_consensus.cc:3037] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.576619  7589 raft_consensus.cc:2454] T 723c99fd647142049bee8313ac4578a1 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:51.576619  7065 raft_consensus.cc:2454] T 723c99fd647142049bee8313ac4578a1 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:51.577031  7262 leader_election.cc:302] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:51.577314  8354 raft_consensus.cc:2781] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.577718  8354 raft_consensus.cc:686] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:51.578019  8354 consensus_queue.cc:237] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.581001  6788 catalog_manager.cc:5484] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:51.670442  8354 raft_consensus.cc:482] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.670698  8354 raft_consensus.cc:504] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.671308  8354 leader_election.cc:288] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:51.672042  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf47308a2aa468f9aab6bbbf66c2a22" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:51.672045  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf47308a2aa468f9aab6bbbf66c2a22" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:51.672284  7589 raft_consensus.cc:2454] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:51.672284  7065 raft_consensus.cc:2454] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:51.672691  7262 leader_election.cc:302] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:51.672959  8354 raft_consensus.cc:2781] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:51.673063  8354 raft_consensus.cc:482] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:51.673183  8354 raft_consensus.cc:3037] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.675298  8354 raft_consensus.cc:504] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.675815  8354 leader_election.cc:288] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195), 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909)
I20240502 08:16:51.676245  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf47308a2aa468f9aab6bbbf66c2a22" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:51.676414  7065 raft_consensus.cc:3037] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.676340  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf47308a2aa468f9aab6bbbf66c2a22" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:51.676553  7589 raft_consensus.cc:3037] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.678428  7065 raft_consensus.cc:2454] T acf47308a2aa468f9aab6bbbf66c2a22 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:51.678617  7589 raft_consensus.cc:2454] T acf47308a2aa468f9aab6bbbf66c2a22 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:51.678858  7262 leader_election.cc:302] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:51.679116  8354 raft_consensus.cc:2781] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.679255  8354 raft_consensus.cc:686] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:51.679478  8354 consensus_queue.cc:237] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } }
I20240502 08:16:51.682507  6788 catalog_manager.cc:5484] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:51.722344  8275 raft_consensus.cc:482] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:51.722522  8275 raft_consensus.cc:504] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.723237  8275 leader_election.cc:288] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:51.723716  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343926b3f14cfa9c3beb3de20eb35b" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:51.723775  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343926b3f14cfa9c3beb3de20eb35b" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:51.723928  7457 raft_consensus.cc:2454] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:51.724043  6932 raft_consensus.cc:2454] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:51.724382  7130 leader_election.cc:302] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:51.724665  8275 raft_consensus.cc:2781] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:51.724764  8275 raft_consensus.cc:482] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:51.724858  8275 raft_consensus.cc:3037] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.727196  8275 raft_consensus.cc:504] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.727741  8275 leader_election.cc:288] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359)
I20240502 08:16:51.728188  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343926b3f14cfa9c3beb3de20eb35b" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:51.728186  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53343926b3f14cfa9c3beb3de20eb35b" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:51.728413  6932 raft_consensus.cc:3037] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.728412  7457 raft_consensus.cc:3037] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:51.731206  7457 raft_consensus.cc:2454] T 53343926b3f14cfa9c3beb3de20eb35b P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:51.731204  6932 raft_consensus.cc:2454] T 53343926b3f14cfa9c3beb3de20eb35b P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:51.731585  7130 leader_election.cc:302] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:51.731880  8275 raft_consensus.cc:2781] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:51.732280  8275 raft_consensus.cc:686] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [term 1 LEADER]: Becoming Leader. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Running, Role: LEADER
I20240502 08:16:51.732543  8275 consensus_queue.cc:237] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } }
I20240502 08:16:51.735474  6788 catalog_manager.cc:5484] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 136680942cc947b0b19be2f50bb114eb (127.6.151.67). New cstate: current_term: 1 leader_uuid: "136680942cc947b0b19be2f50bb114eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:52.026110  8354 consensus_queue.cc:1035] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.046581  8354 consensus_queue.cc:1035] T 723c99fd647142049bee8313ac4578a1 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:52.140954  8531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:52.141314  8531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:16:52.148361  8275 consensus_queue.cc:1035] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:52.151162  8531 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:52.170735  8275 consensus_queue.cc:1035] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.182690  8354 consensus_queue.cc:1035] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:52.192478  8354 consensus_queue.cc:1035] T acf47308a2aa468f9aab6bbbf66c2a22 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'

INSERT report
    rows total: 2000
    time total: 255.161 ms
  time per row: 0.127581 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:16:53.537782  6792 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_10"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:53.546872  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 990eeefc376b47d5ad1854284f0e19a1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.547065  6912 tablet_service.cc:1467] Processing CreateTablet for tablet 990eeefc376b47d5ad1854284f0e19a1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.547384  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990eeefc376b47d5ad1854284f0e19a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.547502  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990eeefc376b47d5ad1854284f0e19a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.548187  7569 tablet_service.cc:1467] Processing CreateTablet for tablet bf420c4ffdd44cebbf195a418a779d8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.548597  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf420c4ffdd44cebbf195a418a779d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.550683  7306 tablet_service.cc:1467] Processing CreateTablet for tablet bf420c4ffdd44cebbf195a418a779d8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.551126  7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf420c4ffdd44cebbf195a418a779d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.559794  7175 tablet_service.cc:1467] Processing CreateTablet for tablet 990eeefc376b47d5ad1854284f0e19a1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.560035  7044 tablet_service.cc:1467] Processing CreateTablet for tablet bf420c4ffdd44cebbf195a418a779d8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.560241  7175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990eeefc376b47d5ad1854284f0e19a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.560433  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf420c4ffdd44cebbf195a418a779d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.561148  7438 tablet_service.cc:1467] Processing CreateTablet for tablet c7ea91f9f9f647a48db2d07dbe25e7aa (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.561549  7438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ea91f9f9f647a48db2d07dbe25e7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.561885  7176 tablet_service.cc:1467] Processing CreateTablet for tablet c7ea91f9f9f647a48db2d07dbe25e7aa (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.562280  7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ea91f9f9f647a48db2d07dbe25e7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.567056  6913 tablet_service.cc:1467] Processing CreateTablet for tablet c7ea91f9f9f647a48db2d07dbe25e7aa (DEFAULT_TABLE table=rebalance_test_table_10 [id=d88a968c60fe4ef392778d44b0a745cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:53.567476  6913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ea91f9f9f647a48db2d07dbe25e7aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:53.571183  8602 tablet_bootstrap.cc:492] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:53.573292  8603 tablet_bootstrap.cc:492] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:53.575549  8602 tablet_bootstrap.cc:654] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.575852  8603 tablet_bootstrap.cc:654] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.576081  8604 tablet_bootstrap.cc:492] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:53.577569  8601 tablet_bootstrap.cc:492] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:53.577991  8604 tablet_bootstrap.cc:654] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.579819  8601 tablet_bootstrap.cc:654] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.582062  8603 tablet_bootstrap.cc:492] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:53.582312  8603 ts_tablet_manager.cc:1367] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.000s
I20240502 08:16:53.585542  8603 raft_consensus.cc:348] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.585916  8603 raft_consensus.cc:374] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.586614  8603 raft_consensus.cc:729] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.587088  8603 consensus_queue.cc:260] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.588145  8603 ts_tablet_manager.cc:1398] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.002s
I20240502 08:16:53.594300  8604 tablet_bootstrap.cc:492] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:53.594565  8604 ts_tablet_manager.cc:1367] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.000s
I20240502 08:16:53.595997  8609 tablet_bootstrap.cc:492] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:53.595983  8604 raft_consensus.cc:348] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.596210  8604 raft_consensus.cc:374] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.596314  8604 raft_consensus.cc:729] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.596552  8604 consensus_queue.cc:260] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.597060  8604 ts_tablet_manager.cc:1398] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:16:53.602442  8609 tablet_bootstrap.cc:654] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.608315  8600 tablet_bootstrap.cc:492] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:53.609529  8602 tablet_bootstrap.cc:492] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:53.609714  8602 ts_tablet_manager.cc:1367] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.006s
I20240502 08:16:53.610109  8604 tablet_bootstrap.cc:492] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:53.610509  8602 raft_consensus.cc:348] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.610781  8602 raft_consensus.cc:374] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.610905  8602 raft_consensus.cc:729] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.611186  8602 consensus_queue.cc:260] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.611822  8602 ts_tablet_manager.cc:1398] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:53.613957  8604 tablet_bootstrap.cc:654] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.614280  8600 tablet_bootstrap.cc:654] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.617375  8609 tablet_bootstrap.cc:492] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:53.617606  8609 ts_tablet_manager.cc:1367] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.022s	user 0.005s	sys 0.000s
I20240502 08:16:53.618795  8609 raft_consensus.cc:348] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.619048  8609 raft_consensus.cc:374] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.619210  8609 raft_consensus.cc:729] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.619488  8609 consensus_queue.cc:260] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.620539  8609 ts_tablet_manager.cc:1398] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:53.620820  8601 tablet_bootstrap.cc:492] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:53.620959  8601 ts_tablet_manager.cc:1367] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.044s	user 0.003s	sys 0.006s
I20240502 08:16:53.621356  8600 tablet_bootstrap.cc:492] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:53.621549  8600 ts_tablet_manager.cc:1367] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.000s
I20240502 08:16:53.621982  8601 raft_consensus.cc:348] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.622249  8601 raft_consensus.cc:374] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.622347  8601 raft_consensus.cc:729] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.622578  8601 consensus_queue.cc:260] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.622519  8600 raft_consensus.cc:348] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.622798  8600 raft_consensus.cc:374] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.622900  8600 raft_consensus.cc:729] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.623220  8600 consensus_queue.cc:260] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.624109  8604 tablet_bootstrap.cc:492] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:53.624337  8604 ts_tablet_manager.cc:1367] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.014s	user 0.002s	sys 0.001s
I20240502 08:16:53.624641  8600 ts_tablet_manager.cc:1398] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:16:53.624895  8600 tablet_bootstrap.cc:492] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:53.625200  8604 raft_consensus.cc:348] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.625420  8604 raft_consensus.cc:374] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.625492  8604 raft_consensus.cc:729] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.625790  8604 consensus_queue.cc:260] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.626556  8600 tablet_bootstrap.cc:654] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.626664  8604 ts_tablet_manager.cc:1398] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:16:53.627027  8601 ts_tablet_manager.cc:1398] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 08:16:53.630977  8600 tablet_bootstrap.cc:492] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:53.631338  8600 ts_tablet_manager.cc:1367] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.006s	user 0.005s	sys 0.000s
I20240502 08:16:53.632015  8600 raft_consensus.cc:348] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.632232  8600 raft_consensus.cc:374] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.632318  8600 raft_consensus.cc:729] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.632570  8600 consensus_queue.cc:260] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.632866  8601 tablet_bootstrap.cc:492] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:53.633217  8600 ts_tablet_manager.cc:1398] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:16:53.634696  8601 tablet_bootstrap.cc:654] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:53.636569  8601 tablet_bootstrap.cc:492] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:53.636776  8601 ts_tablet_manager.cc:1367] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 08:16:53.637590  8601 raft_consensus.cc:348] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.637838  8601 raft_consensus.cc:374] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:53.637913  8601 raft_consensus.cc:729] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:53.638157  8601 consensus_queue.cc:260] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.638598  8601 ts_tablet_manager.cc:1398] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:16:53.784718  8593 raft_consensus.cc:482] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.784945  8593 raft_consensus.cc:504] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.785923  8593 leader_election.cc:288] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:53.786424  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ea91f9f9f647a48db2d07dbe25e7aa" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:53.786494  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ea91f9f9f647a48db2d07dbe25e7aa" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:53.786692  6932 raft_consensus.cc:2454] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:53.786746  7195 raft_consensus.cc:2454] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 0.
I20240502 08:16:53.787123  7391 leader_election.cc:302] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:53.787389  8593 raft_consensus.cc:2781] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:53.787515  8593 raft_consensus.cc:482] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:53.787611  8593 raft_consensus.cc:3037] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.789585  8593 raft_consensus.cc:504] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.790136  8593 leader_election.cc:288] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Requested vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:53.790591  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ea91f9f9f647a48db2d07dbe25e7aa" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:53.790782  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ea91f9f9f647a48db2d07dbe25e7aa" candidate_uuid: "a5b21a0f245140b293e607e8f554fa1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:53.790849  6932 raft_consensus.cc:3037] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.791009  7195 raft_consensus.cc:3037] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.793761  7195 raft_consensus.cc:2454] T c7ea91f9f9f647a48db2d07dbe25e7aa P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:53.793774  6932 raft_consensus.cc:2454] T c7ea91f9f9f647a48db2d07dbe25e7aa P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5b21a0f245140b293e607e8f554fa1f in term 1.
I20240502 08:16:53.794147  7391 leader_election.cc:302] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, a5b21a0f245140b293e607e8f554fa1f; no voters: 
I20240502 08:16:53.794487  8593 raft_consensus.cc:2781] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:53.794862  8593 raft_consensus.cc:686] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [term 1 LEADER]: Becoming Leader. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Running, Role: LEADER
I20240502 08:16:53.795150  8593 consensus_queue.cc:237] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.798257  6788 catalog_manager.cc:5484] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f reported cstate change: term changed from 0 to 1, leader changed from <none> to a5b21a0f245140b293e607e8f554fa1f (127.6.151.69). New cstate: current_term: 1 leader_uuid: "a5b21a0f245140b293e607e8f554fa1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:53.857021  8595 raft_consensus.cc:482] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.857228  8595 raft_consensus.cc:504] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.858201  8595 leader_election.cc:288] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:53.858815  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990eeefc376b47d5ad1854284f0e19a1" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb" is_pre_election: true
I20240502 08:16:53.858773  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990eeefc376b47d5ad1854284f0e19a1" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:53.859045  7195 raft_consensus.cc:2454] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:53.859053  7457 raft_consensus.cc:2454] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 0.
I20240502 08:16:53.859591  6867 leader_election.cc:302] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:53.859966  8610 raft_consensus.cc:2781] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:53.860073  8610 raft_consensus.cc:482] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:53.860231  8610 raft_consensus.cc:3037] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.862484  8610 raft_consensus.cc:504] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.862555  8418 raft_consensus.cc:482] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:53.862727  8418 raft_consensus.cc:504] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.863176  8610 leader_election.cc:288] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Requested vote from peers a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083), 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791)
I20240502 08:16:53.863394  8418 leader_election.cc:288] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:53.863788  7195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990eeefc376b47d5ad1854284f0e19a1" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136680942cc947b0b19be2f50bb114eb"
I20240502 08:16:53.863991  7195 raft_consensus.cc:3037] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.863925  7588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf420c4ffdd44cebbf195a418a779d8a" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:53.863955  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf420c4ffdd44cebbf195a418a779d8a" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" is_pre_election: true
I20240502 08:16:53.864173  7326 raft_consensus.cc:2454] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:53.864178  7588 raft_consensus.cc:2454] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 0.
I20240502 08:16:53.864347  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990eeefc376b47d5ad1854284f0e19a1" candidate_uuid: "cc93fe73601544fe8a8aa013f8de095b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:53.864550  7457 raft_consensus.cc:3037] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.864763  7000 leader_election.cc:302] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881e59a197394322ade814528d5e2cdf, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:53.865070  8418 raft_consensus.cc:2781] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:53.865181  8418 raft_consensus.cc:482] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:53.865260  8418 raft_consensus.cc:3037] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.866852  7195 raft_consensus.cc:2454] T 990eeefc376b47d5ad1854284f0e19a1 P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:53.866856  7457 raft_consensus.cc:2454] T 990eeefc376b47d5ad1854284f0e19a1 P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc93fe73601544fe8a8aa013f8de095b in term 1.
I20240502 08:16:53.867314  6867 leader_election.cc:302] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5b21a0f245140b293e607e8f554fa1f, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:53.867664  8610 raft_consensus.cc:2781] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:53.867872  8610 raft_consensus.cc:686] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [term 1 LEADER]: Becoming Leader. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Running, Role: LEADER
I20240502 08:16:53.868165  8610 consensus_queue.cc:237] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } }
I20240502 08:16:53.868599  8418 raft_consensus.cc:504] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.869657  7588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf420c4ffdd44cebbf195a418a779d8a" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:53.869665  7326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf420c4ffdd44cebbf195a418a779d8a" candidate_uuid: "b8432324fa494be3a41163adc25b0865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3"
I20240502 08:16:53.869858  7588 raft_consensus.cc:3037] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.869876  7326 raft_consensus.cc:3037] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:53.870628  8418 leader_election.cc:288] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Requested vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071)
I20240502 08:16:53.872045  7326 raft_consensus.cc:2454] T bf420c4ffdd44cebbf195a418a779d8a P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:53.871868  6788 catalog_manager.cc:5484] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b reported cstate change: term changed from 0 to 1, leader changed from <none> to cc93fe73601544fe8a8aa013f8de095b (127.6.151.65). New cstate: current_term: 1 leader_uuid: "cc93fe73601544fe8a8aa013f8de095b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:53.872443  7588 raft_consensus.cc:2454] T bf420c4ffdd44cebbf195a418a779d8a P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8432324fa494be3a41163adc25b0865 in term 1.
I20240502 08:16:53.872879  7001 leader_election.cc:302] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:53.873159  8418 raft_consensus.cc:2781] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:53.873394  8418 raft_consensus.cc:686] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [term 1 LEADER]: Becoming Leader. State: Replica: b8432324fa494be3a41163adc25b0865, State: Running, Role: LEADER
I20240502 08:16:53.873744  8418 consensus_queue.cc:237] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } }
I20240502 08:16:53.876857  6788 catalog_manager.cc:5484] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8432324fa494be3a41163adc25b0865 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "b8432324fa494be3a41163adc25b0865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:54.199339  8593 consensus_queue.cc:1035] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.206996  8616 consensus_queue.cc:1035] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:54.285655  8621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:54.285979  8621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:54.294406  8621 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:16:54.295501  8610 consensus_queue.cc:1035] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.299990  8631 consensus_queue.cc:1035] T 990eeefc376b47d5ad1854284f0e19a1 P cc93fe73601544fe8a8aa013f8de095b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
I20240502 08:16:54.390882  8418 consensus_queue.cc:1035] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:54.428385  8653 consensus_queue.cc:1035] T bf420c4ffdd44cebbf195a418a779d8a P b8432324fa494be3a41163adc25b0865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 307.738 ms
  time per row: 0.153869 ms
I20240502 08:16:54.756822  6792 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48120:
name: "rebalance_test_table_11"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:16:54.766069  7044 tablet_service.cc:1467] Processing CreateTablet for tablet ce012d37eaba46af8f68061c9a9b1bb7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.766578  7044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce012d37eaba46af8f68061c9a9b1bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.766927  7568 tablet_service.cc:1467] Processing CreateTablet for tablet 34569d2e82bb4c95bd852b0b2d58d652 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.766966  7306 tablet_service.cc:1467] Processing CreateTablet for tablet ce012d37eaba46af8f68061c9a9b1bb7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.767359  7568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34569d2e82bb4c95bd852b0b2d58d652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.767419  7306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce012d37eaba46af8f68061c9a9b1bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.767469  7569 tablet_service.cc:1467] Processing CreateTablet for tablet ce012d37eaba46af8f68061c9a9b1bb7 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.767822  7569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce012d37eaba46af8f68061c9a9b1bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.769626  7045 tablet_service.cc:1467] Processing CreateTablet for tablet 34569d2e82bb4c95bd852b0b2d58d652 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.769989  7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34569d2e82bb4c95bd852b0b2d58d652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.771123  7176 tablet_service.cc:1467] Processing CreateTablet for tablet 014ad747feed43c1b8988a9ab73b87ca (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.771507  7176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014ad747feed43c1b8988a9ab73b87ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.773783  7307 tablet_service.cc:1467] Processing CreateTablet for tablet 34569d2e82bb4c95bd852b0b2d58d652 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.774112  7307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34569d2e82bb4c95bd852b0b2d58d652. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.776618  6912 tablet_service.cc:1467] Processing CreateTablet for tablet 014ad747feed43c1b8988a9ab73b87ca (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.777079  6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014ad747feed43c1b8988a9ab73b87ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.780704  7437 tablet_service.cc:1467] Processing CreateTablet for tablet 014ad747feed43c1b8988a9ab73b87ca (DEFAULT_TABLE table=rebalance_test_table_11 [id=6d8fb78b0bcd45e5bcf8c1ad4c38faa1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:16:54.781116  7437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 014ad747feed43c1b8988a9ab73b87ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:16:54.789988  8694 tablet_bootstrap.cc:492] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:54.791389  8691 tablet_bootstrap.cc:492] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb: Bootstrap starting.
I20240502 08:16:54.792369  8690 tablet_bootstrap.cc:492] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:54.792944  8692 tablet_bootstrap.cc:492] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:54.793876  8694 tablet_bootstrap.cc:654] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.795295  8693 tablet_bootstrap.cc:492] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b: Bootstrap starting.
I20240502 08:16:54.796345  8690 tablet_bootstrap.cc:654] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.796450  8691 tablet_bootstrap.cc:654] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.797775  8695 tablet_bootstrap.cc:492] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f: Bootstrap starting.
I20240502 08:16:54.797804  8692 tablet_bootstrap.cc:654] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.801317  8693 tablet_bootstrap.cc:654] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.802999  8695 tablet_bootstrap.cc:654] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.804373  8692 tablet_bootstrap.cc:492] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:54.804642  8692 ts_tablet_manager.cc:1367] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.002s
I20240502 08:16:54.806252  8692 raft_consensus.cc:348] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.806722  8692 raft_consensus.cc:374] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.806898  8692 raft_consensus.cc:729] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.807247  8692 consensus_queue.cc:260] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.807813  8692 ts_tablet_manager.cc:1398] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:16:54.808271  8692 tablet_bootstrap.cc:492] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf: Bootstrap starting.
I20240502 08:16:54.808759  8691 tablet_bootstrap.cc:492] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb: No bootstrap required, opened a new log
I20240502 08:16:54.808950  8691 ts_tablet_manager.cc:1367] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb: Time spent bootstrapping tablet: real 0.018s	user 0.007s	sys 0.000s
I20240502 08:16:54.809922  8691 raft_consensus.cc:348] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.810175  8691 raft_consensus.cc:374] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.810281  8691 raft_consensus.cc:729] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.810571  8691 consensus_queue.cc:260] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.811964  8692 tablet_bootstrap.cc:654] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.813220  8691 ts_tablet_manager.cc:1398] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:16:54.814157  8695 tablet_bootstrap.cc:492] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f: No bootstrap required, opened a new log
I20240502 08:16:54.814368  8695 ts_tablet_manager.cc:1367] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.000s
I20240502 08:16:54.814535  8693 tablet_bootstrap.cc:492] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b: No bootstrap required, opened a new log
I20240502 08:16:54.814777  8693 ts_tablet_manager.cc:1367] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b: Time spent bootstrapping tablet: real 0.020s	user 0.002s	sys 0.005s
I20240502 08:16:54.815256  8695 raft_consensus.cc:348] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.815476  8695 raft_consensus.cc:374] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.815572  8695 raft_consensus.cc:729] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b21a0f245140b293e607e8f554fa1f, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.815685  8693 raft_consensus.cc:348] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.815889  8693 raft_consensus.cc:374] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.815794  8695 consensus_queue.cc:260] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.815980  8693 raft_consensus.cc:729] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc93fe73601544fe8a8aa013f8de095b, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.816211  8693 consensus_queue.cc:260] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.816349  8695 ts_tablet_manager.cc:1398] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:16:54.816854  8693 ts_tablet_manager.cc:1398] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:16:54.817973  8694 tablet_bootstrap.cc:492] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:54.818140  8694 ts_tablet_manager.cc:1367] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.012s
I20240502 08:16:54.819799  8690 tablet_bootstrap.cc:492] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:54.820026  8690 ts_tablet_manager.cc:1367] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.028s	user 0.005s	sys 0.005s
I20240502 08:16:54.820951  8692 tablet_bootstrap.cc:492] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf: No bootstrap required, opened a new log
I20240502 08:16:54.820938  8690 raft_consensus.cc:348] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.821193  8690 raft_consensus.cc:374] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.821202  8692 ts_tablet_manager.cc:1367] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.001s
I20240502 08:16:54.821321  8690 raft_consensus.cc:729] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.821609  8690 consensus_queue.cc:260] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.821593  8694 raft_consensus.cc:348] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.821785  8694 raft_consensus.cc:374] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.821920  8694 raft_consensus.cc:729] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.822217  8690 ts_tablet_manager.cc:1398] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:16:54.822216  8694 consensus_queue.cc:260] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.822571  8690 tablet_bootstrap.cc:492] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3: Bootstrap starting.
I20240502 08:16:54.823321  8692 raft_consensus.cc:348] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.823853  8692 raft_consensus.cc:374] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.823980  8692 raft_consensus.cc:729] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881e59a197394322ade814528d5e2cdf, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.824214  8690 tablet_bootstrap.cc:654] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.824472  8692 consensus_queue.cc:260] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.824685  8694 ts_tablet_manager.cc:1398] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 08:16:54.824885  8580 raft_consensus.cc:482] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.825050  8694 tablet_bootstrap.cc:492] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865: Bootstrap starting.
I20240502 08:16:54.825050  8580 raft_consensus.cc:504] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.825863  8580 leader_election.cc:288] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083)
I20240502 08:16:54.826697  7457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014ad747feed43c1b8988a9ab73b87ca" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f" is_pre_election: true
I20240502 08:16:54.826965  7457 raft_consensus.cc:2454] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:54.827581  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014ad747feed43c1b8988a9ab73b87ca" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b" is_pre_election: true
I20240502 08:16:54.827821  6932 raft_consensus.cc:2454] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 0.
I20240502 08:16:54.828524  7129 leader_election.cc:302] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:54.829563  8694 tablet_bootstrap.cc:654] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865: Neither blocks nor log segments found. Creating new log.
I20240502 08:16:54.830801  8580 raft_consensus.cc:2781] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.830910  8580 raft_consensus.cc:482] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.830991  8580 raft_consensus.cc:3037] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.832474  8690 tablet_bootstrap.cc:492] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3: No bootstrap required, opened a new log
I20240502 08:16:54.832626  8690 ts_tablet_manager.cc:1367] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.005s
I20240502 08:16:54.833768  8580 raft_consensus.cc:504] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.835059  6932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014ad747feed43c1b8988a9ab73b87ca" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc93fe73601544fe8a8aa013f8de095b"
I20240502 08:16:54.835297  6932 raft_consensus.cc:3037] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.835207  8690 raft_consensus.cc:348] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.835415  8690 raft_consensus.cc:374] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.835481  8690 raft_consensus.cc:729] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.836166  8690 consensus_queue.cc:260] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.837014  8690 ts_tablet_manager.cc:1398] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.002s
I20240502 08:16:54.837492  8354 raft_consensus.cc:482] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.837512  8580 leader_election.cc:288] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Requested vote from peers cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359), a5b21a0f245140b293e607e8f554fa1f (127.6.151.69:44083)
I20240502 08:16:54.837622  8354 raft_consensus.cc:504] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.837890  8694 tablet_bootstrap.cc:492] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865: No bootstrap required, opened a new log
I20240502 08:16:54.837961  6932 raft_consensus.cc:2454] T 014ad747feed43c1b8988a9ab73b87ca P cc93fe73601544fe8a8aa013f8de095b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:54.838048  8694 ts_tablet_manager.cc:1367] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.004s
I20240502 08:16:54.838330  8354 leader_election.cc:288] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:54.838778  8694 raft_consensus.cc:348] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.839017  8694 raft_consensus.cc:374] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:16:54.839089  8694 raft_consensus.cc:729] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8432324fa494be3a41163adc25b0865, State: Initialized, Role: FOLLOWER
I20240502 08:16:54.839198  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34569d2e82bb4c95bd852b0b2d58d652" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:54.839311  8694 consensus_queue.cc:260] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.839429  7589 raft_consensus.cc:2454] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:54.839771  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34569d2e82bb4c95bd852b0b2d58d652" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:54.839864  8694 ts_tablet_manager.cc:1398] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:16:54.839987  7065 raft_consensus.cc:2454] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:54.840426  7262 leader_election.cc:302] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:54.840602  7129 leader_election.cc:302] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136680942cc947b0b19be2f50bb114eb, cc93fe73601544fe8a8aa013f8de095b; no voters: 
I20240502 08:16:54.840708  8354 raft_consensus.cc:2781] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.840806  8354 raft_consensus.cc:482] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.840878  8354 raft_consensus.cc:3037] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.841238  7458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "014ad747feed43c1b8988a9ab73b87ca" candidate_uuid: "136680942cc947b0b19be2f50bb114eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b21a0f245140b293e607e8f554fa1f"
I20240502 08:16:54.841432  7458 raft_consensus.cc:3037] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.843575  7458 raft_consensus.cc:2454] T 014ad747feed43c1b8988a9ab73b87ca P a5b21a0f245140b293e607e8f554fa1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 136680942cc947b0b19be2f50bb114eb in term 1.
I20240502 08:16:54.843820  8354 raft_consensus.cc:504] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.844616  8354 leader_election.cc:288] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:54.844934  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34569d2e82bb4c95bd852b0b2d58d652" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:54.845145  7065 raft_consensus.cc:3037] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.845575  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34569d2e82bb4c95bd852b0b2d58d652" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:54.845808  7589 raft_consensus.cc:3037] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.848060  7065 raft_consensus.cc:2454] T 34569d2e82bb4c95bd852b0b2d58d652 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:54.848569  7262 leader_election.cc:302] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:54.849443  8354 raft_consensus.cc:2781] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.851871  7589 raft_consensus.cc:2454] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:54.852792  8580 raft_consensus.cc:2781] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.854702  8692 ts_tablet_manager.cc:1398] T 34569d2e82bb4c95bd852b0b2d58d652 P 881e59a197394322ade814528d5e2cdf: Time spent starting tablet: real 0.033s	user 0.001s	sys 0.001s
I20240502 08:16:54.862907  8354 raft_consensus.cc:686] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:54.863273  8354 consensus_queue.cc:237] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.879190  8580 raft_consensus.cc:686] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [term 1 LEADER]: Becoming Leader. State: Replica: 136680942cc947b0b19be2f50bb114eb, State: Running, Role: LEADER
I20240502 08:16:54.879781  8580 consensus_queue.cc:237] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } }
I20240502 08:16:54.885747  6786 catalog_manager.cc:5484] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:54.888852  6784 catalog_manager.cc:5484] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 136680942cc947b0b19be2f50bb114eb (127.6.151.67). New cstate: current_term: 1 leader_uuid: "136680942cc947b0b19be2f50bb114eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "136680942cc947b0b19be2f50bb114eb" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36791 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:54.901324  8354 raft_consensus.cc:482] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:16:54.901562  8354 raft_consensus.cc:504] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.902148  8354 leader_election.cc:288] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:54.902575  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce012d37eaba46af8f68061c9a9b1bb7" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf" is_pre_election: true
I20240502 08:16:54.902700  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce012d37eaba46af8f68061c9a9b1bb7" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865" is_pre_election: true
I20240502 08:16:54.902799  7589 raft_consensus.cc:2454] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:54.902906  7065 raft_consensus.cc:2454] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 0.
I20240502 08:16:54.903201  7262 leader_election.cc:302] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, 881e59a197394322ade814528d5e2cdf; no voters: 
I20240502 08:16:54.903426  8354 raft_consensus.cc:2781] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:16:54.903522  8354 raft_consensus.cc:482] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:16:54.903628  8354 raft_consensus.cc:3037] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.905434  8354 raft_consensus.cc:504] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.905967  8354 leader_election.cc:288] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Requested vote from peers 881e59a197394322ade814528d5e2cdf (127.6.151.70:36909), b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195)
I20240502 08:16:54.906317  7589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce012d37eaba46af8f68061c9a9b1bb7" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881e59a197394322ade814528d5e2cdf"
I20240502 08:16:54.906404  7065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce012d37eaba46af8f68061c9a9b1bb7" candidate_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8432324fa494be3a41163adc25b0865"
I20240502 08:16:54.906522  7589 raft_consensus.cc:3037] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.906580  7065 raft_consensus.cc:3037] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:16:54.909276  7065 raft_consensus.cc:2454] T ce012d37eaba46af8f68061c9a9b1bb7 P b8432324fa494be3a41163adc25b0865 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:54.909271  7589 raft_consensus.cc:2454] T ce012d37eaba46af8f68061c9a9b1bb7 P 881e59a197394322ade814528d5e2cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52b3f590e1b14b14ad7d3b827e9b21f3 in term 1.
I20240502 08:16:54.909603  7262 leader_election.cc:302] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52b3f590e1b14b14ad7d3b827e9b21f3, b8432324fa494be3a41163adc25b0865; no voters: 
I20240502 08:16:54.909833  8354 raft_consensus.cc:2781] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:16:54.909991  8354 raft_consensus.cc:686] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [term 1 LEADER]: Becoming Leader. State: Replica: 52b3f590e1b14b14ad7d3b827e9b21f3, State: Running, Role: LEADER
I20240502 08:16:54.910254  8354 consensus_queue.cc:237] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } }
I20240502 08:16:54.913355  6784 catalog_manager.cc:5484] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52b3f590e1b14b14ad7d3b827e9b21f3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 } health_report { overall_health: UNKNOWN } } }
I20240502 08:16:55.284790  8580 consensus_queue.cc:1035] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b21a0f245140b293e607e8f554fa1f" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.294052  8580 consensus_queue.cc:1035] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc93fe73601544fe8a8aa013f8de095b" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:16:55.305163  8714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:16:55.305516  8714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:16:55.313215  8714 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 08:16:55.374848  8354 consensus_queue.cc:1035] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.413918  8709 consensus_queue.cc:1035] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.419202  8354 consensus_queue.cc:1035] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8432324fa494be3a41163adc25b0865" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 46195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:16:55.419548  8745 consensus_queue.cc:1035] T 34569d2e82bb4c95bd852b0b2d58d652 P 52b3f590e1b14b14ad7d3b827e9b21f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881e59a197394322ade814528d5e2cdf" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 36909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 306.473 ms
  time per row: 0.153237 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:16:57.277196  7176 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:57.294937  7306 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:16:57.310699  7569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:57.316696  7437 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:57.344542  7045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:16:57.367555  6912 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:57.600777  7176 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:57.616344  7306 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:16:57.626986  7437 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:57.634343  7045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:16:57.650157  7569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:57.676461  6912 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:57.903191  7176 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:57.903661  7306 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:16:57.923694  7569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:57.941284  7437 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:57.963160  7045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:16:57.994851  6912 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.217399  7176 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.230814  7569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.230825  7437 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.244439  7045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:16:58.247784  7306 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:16:58.250120  6912 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.522975  7176 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.531397  7306 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:16:58.548156  7569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.552704  7437 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.567576  6912 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.570194  7045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:16:58.964716  7306 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:16:58.969128  7176 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.979260  7569 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:16:58.989497  7437 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:58.993986  6912 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:16:59.008386  7045 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 a10cf1e9c4d74698a44f15d67b76e040 | 127.6.151.126:34373 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                    Value                                                                                     |         Tags          |         Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                          | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                         | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                            | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                         | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                         | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                         | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                           | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                          | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:40585 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 136680942cc947b0b19be2f50bb114eb | 127.6.151.67:36791 | HEALTHY | /B       |       4        |       0
 52b3f590e1b14b14ad7d3b827e9b21f3 | 127.6.151.68:34071 | HEALTHY | /B       |       9        |       0
 881e59a197394322ade814528d5e2cdf | 127.6.151.70:36909 | HEALTHY | /C       |       4        |       0
 a5b21a0f245140b293e607e8f554fa1f | 127.6.151.69:44083 | HEALTHY | /C       |       7        |       0
 b8432324fa494be3a41163adc25b0865 | 127.6.151.66:46195 | HEALTHY | /A       |       5        |       0
 cc93fe73601544fe8a8aa013f8de095b | 127.6.151.65:40359 | HEALTHY | /A       |       7        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 /A       |       2
 /B       |       2
 /C       |       2

Unusual flags for Tablet Server:
                   Flag                   |                                                                                  Value                                                                                   |         Tags          |      Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                      | experimental          | all 6 server(s) checked
 local_ip_for_outbound_sockets            | 127.6.151.65                                                                                                                                                             | experimental          | 127.6.151.65:40359
 local_ip_for_outbound_sockets            | 127.6.151.66                                                                                                                                                             | experimental          | 127.6.151.66:46195
 local_ip_for_outbound_sockets            | 127.6.151.67                                                                                                                                                             | experimental          | 127.6.151.67:36791
 local_ip_for_outbound_sockets            | 127.6.151.68                                                                                                                                                             | experimental          | 127.6.151.68:34071
 local_ip_for_outbound_sockets            | 127.6.151.69                                                                                                                                                             | experimental          | 127.6.151.69:44083
 local_ip_for_outbound_sockets            | 127.6.151.70                                                                                                                                                             | experimental          | 127.6.151.70:36909
 never_fsync                              | true                                                                                                                                                                     | unsafe,advanced       | all 6 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                        | unsafe,hidden         | all 6 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                     | experimental,advanced | all 6 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                     | unsafe                | all 6 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                       | hidden                | all 6 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.6.151.65:40359
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.6.151.66:46195
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.6.151.67:36791
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.6.151.68:34071
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.6.151.69:44083
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.6.151.70:36909

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:40585 | all 6 server(s) checked
 time_source         | builtin            | all 6 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 7 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name           | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
 rebalance_test_table_0  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_10 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_11 | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_5  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_6  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_7  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_8  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_9  | 3  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 18
 First Quartile | 18
 Median         | 18
 Third Quartile | 18
 Maximum        | 18

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 6
 Tables         | 12
 Tablets        | 36
 Replicas       | 108

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 08:16:59.096494  6749 log_verifier.cc:126] Checking tablet 014ad747feed43c1b8988a9ab73b87ca
I20240502 08:16:59.136771  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 014ad747feed43c1b8988a9ab73b87ca
I20240502 08:16:59.137005  6749 log_verifier.cc:126] Checking tablet 030c13ba7de146589cd58a8d3ea774de
I20240502 08:16:59.175251  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 030c13ba7de146589cd58a8d3ea774de
I20240502 08:16:59.175381  6749 log_verifier.cc:126] Checking tablet 05abbc25f6f6470590713c894affd936
I20240502 08:16:59.213235  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 05abbc25f6f6470590713c894affd936
I20240502 08:16:59.213352  6749 log_verifier.cc:126] Checking tablet 0d7648c5b0c442f2aa908e60d584addf
I20240502 08:16:59.261073  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0d7648c5b0c442f2aa908e60d584addf
I20240502 08:16:59.261189  6749 log_verifier.cc:126] Checking tablet 174252fc5fcd4630b409727636aeb2f6
I20240502 08:16:59.304092  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 174252fc5fcd4630b409727636aeb2f6
I20240502 08:16:59.304198  6749 log_verifier.cc:126] Checking tablet 18202248017f434b9fbc36f64a5eb17b
I20240502 08:16:59.347465  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 18202248017f434b9fbc36f64a5eb17b
I20240502 08:16:59.347581  6749 log_verifier.cc:126] Checking tablet 18cf97f591e04ac6bb509d9d258df83a
I20240502 08:16:59.389618  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 18cf97f591e04ac6bb509d9d258df83a
I20240502 08:16:59.389734  6749 log_verifier.cc:126] Checking tablet 19c8e9eca3de4a959109f870f9299dd4
I20240502 08:16:59.430606  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 19c8e9eca3de4a959109f870f9299dd4
I20240502 08:16:59.430774  6749 log_verifier.cc:126] Checking tablet 1b3bec7be4924971bde466a43efaa585
I20240502 08:16:59.481750  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1b3bec7be4924971bde466a43efaa585
I20240502 08:16:59.481873  6749 log_verifier.cc:126] Checking tablet 1e0ba483f04745cbb3522ece53fb2567
I20240502 08:16:59.525700  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1e0ba483f04745cbb3522ece53fb2567
I20240502 08:16:59.525900  6749 log_verifier.cc:126] Checking tablet 2de7d4d111364e048241d0f8b2832241
I20240502 08:16:59.573681  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2de7d4d111364e048241d0f8b2832241
I20240502 08:16:59.573884  6749 log_verifier.cc:126] Checking tablet 34569d2e82bb4c95bd852b0b2d58d652
I20240502 08:16:59.620508  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 34569d2e82bb4c95bd852b0b2d58d652
I20240502 08:16:59.620663  6749 log_verifier.cc:126] Checking tablet 44c06a7143db48fd98fa731b0baa161a
I20240502 08:16:59.664453  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 44c06a7143db48fd98fa731b0baa161a
I20240502 08:16:59.664597  6749 log_verifier.cc:126] Checking tablet 53343926b3f14cfa9c3beb3de20eb35b
I20240502 08:16:59.709041  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 53343926b3f14cfa9c3beb3de20eb35b
I20240502 08:16:59.709149  6749 log_verifier.cc:126] Checking tablet 55bcef87afbf469384cc6e067145fbdc
I20240502 08:16:59.753044  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 55bcef87afbf469384cc6e067145fbdc
I20240502 08:16:59.753163  6749 log_verifier.cc:126] Checking tablet 62c40420104d40e6853ea6fbbf9db21a
I20240502 08:16:59.795428  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 62c40420104d40e6853ea6fbbf9db21a
I20240502 08:16:59.795545  6749 log_verifier.cc:126] Checking tablet 6331a099757442eca1206f464bb354a5
I20240502 08:16:59.840711  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6331a099757442eca1206f464bb354a5
I20240502 08:16:59.840912  6749 log_verifier.cc:126] Checking tablet 723c99fd647142049bee8313ac4578a1
I20240502 08:16:59.889636  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 723c99fd647142049bee8313ac4578a1
I20240502 08:16:59.889757  6749 log_verifier.cc:126] Checking tablet 7c88e7a1a04849d585c5ca9baaed7686
I20240502 08:16:59.938575  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7c88e7a1a04849d585c5ca9baaed7686
I20240502 08:16:59.938730  6749 log_verifier.cc:126] Checking tablet 834e8b656bcb417184b50efbead45f86
I20240502 08:16:59.982185  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 834e8b656bcb417184b50efbead45f86
I20240502 08:16:59.982309  6749 log_verifier.cc:126] Checking tablet 990eeefc376b47d5ad1854284f0e19a1
I20240502 08:17:00.033670  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 990eeefc376b47d5ad1854284f0e19a1
I20240502 08:17:00.033864  6749 log_verifier.cc:126] Checking tablet a27e8d56086e44a99aade2205fa32f18
I20240502 08:17:00.078143  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a27e8d56086e44a99aade2205fa32f18
I20240502 08:17:00.078264  6749 log_verifier.cc:126] Checking tablet acf47308a2aa468f9aab6bbbf66c2a22
I20240502 08:17:00.122489  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet acf47308a2aa468f9aab6bbbf66c2a22
I20240502 08:17:00.122628  6749 log_verifier.cc:126] Checking tablet aec8be7dfe664e29ad5153c154d4f585
I20240502 08:17:00.167372  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet aec8be7dfe664e29ad5153c154d4f585
I20240502 08:17:00.167496  6749 log_verifier.cc:126] Checking tablet b4b502d33f5c45d2a0d80c3588404021
I20240502 08:17:00.210435  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b4b502d33f5c45d2a0d80c3588404021
I20240502 08:17:00.211081  6749 log_verifier.cc:126] Checking tablet bf420c4ffdd44cebbf195a418a779d8a
I20240502 08:17:00.253954  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet bf420c4ffdd44cebbf195a418a779d8a
I20240502 08:17:00.254098  6749 log_verifier.cc:126] Checking tablet c226114065e6429b8fef55539c315388
I20240502 08:17:00.297489  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c226114065e6429b8fef55539c315388
I20240502 08:17:00.297616  6749 log_verifier.cc:126] Checking tablet c7ea91f9f9f647a48db2d07dbe25e7aa
I20240502 08:17:00.342167  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c7ea91f9f9f647a48db2d07dbe25e7aa
I20240502 08:17:00.342296  6749 log_verifier.cc:126] Checking tablet c9282750898d4375a36d4fd0be7008ec
I20240502 08:17:00.386556  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c9282750898d4375a36d4fd0be7008ec
I20240502 08:17:00.386725  6749 log_verifier.cc:126] Checking tablet ce012d37eaba46af8f68061c9a9b1bb7
I20240502 08:17:00.428550  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ce012d37eaba46af8f68061c9a9b1bb7
I20240502 08:17:00.428684  6749 log_verifier.cc:126] Checking tablet d203d388047241268160248093e177a7
I20240502 08:17:00.471894  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d203d388047241268160248093e177a7
I20240502 08:17:00.472046  6749 log_verifier.cc:126] Checking tablet da11fa181227431f9119eba713d32e1a
I20240502 08:17:00.516856  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet da11fa181227431f9119eba713d32e1a
I20240502 08:17:00.516993  6749 log_verifier.cc:126] Checking tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0
I20240502 08:17:00.563171  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dc7de5d2bd3b4518b1c4d7389f7c5fc0
I20240502 08:17:00.563311  6749 log_verifier.cc:126] Checking tablet e233e1477fbb4e1fa5c11a788c56b7e0
I20240502 08:17:00.610424  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e233e1477fbb4e1fa5c11a788c56b7e0
I20240502 08:17:00.610534  6749 log_verifier.cc:126] Checking tablet ec2c667c07b64ac987cdba570dfce4db
I20240502 08:17:00.653040  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ec2c667c07b64ac987cdba570dfce4db
I20240502 08:17:00.653191  6749 log_verifier.cc:126] Checking tablet fdc2c1cbd9c14fddb57a7d9c103fa496
I20240502 08:17:00.700042  6749 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fdc2c1cbd9c14fddb57a7d9c103fa496
I20240502 08:17:00.720391  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 6851
I20240502 08:17:00.732564  6973 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:00.872596  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 6851
I20240502 08:17:00.899014  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 6982
I20240502 08:17:00.904920  7104 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:01.004464  7391 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111)
W20240502 08:17:01.006187  7391 consensus_peers.cc:485] T 174252fc5fcd4630b409727636aeb2f6 P a5b21a0f245140b293e607e8f554fa1f -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.010565  7391 consensus_peers.cc:485] T 18202248017f434b9fbc36f64a5eb17b P a5b21a0f245140b293e607e8f554fa1f -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.036473  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 6982
W20240502 08:17:01.044054  7129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111)
W20240502 08:17:01.044468  7391 consensus_peers.cc:485] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.045472  7129 consensus_peers.cc:485] T 014ad747feed43c1b8988a9ab73b87ca P 136680942cc947b0b19be2f50bb114eb -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.062901  7262 connection.cc:535] client connection to 127.6.151.66:46195 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:01.063210  7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:01.063184  7524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.151.66:46195 (error 108)
I20240502 08:17:01.063565  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 7113
W20240502 08:17:01.065840  7524 consensus_peers.cc:485] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.066008  7524 consensus_peers.cc:485] T ec2c667c07b64ac987cdba570dfce4db P 881e59a197394322ade814528d5e2cdf -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.066069  7262 consensus_peers.cc:485] T 0d7648c5b0c442f2aa908e60d584addf P 52b3f590e1b14b14ad7d3b827e9b21f3 -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.070282  7236 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:01.109179  7524 consensus_peers.cc:485] T 6331a099757442eca1206f464bb354a5 P 881e59a197394322ade814528d5e2cdf -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.114287  7391 consensus_peers.cc:485] T c7ea91f9f9f647a48db2d07dbe25e7aa P a5b21a0f245140b293e607e8f554fa1f -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.135123  7262 consensus_peers.cc:485] T b4b502d33f5c45d2a0d80c3588404021 P 52b3f590e1b14b14ad7d3b827e9b21f3 -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.143746  7262 consensus_peers.cc:485] T ce012d37eaba46af8f68061c9a9b1bb7 P 52b3f590e1b14b14ad7d3b827e9b21f3 -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.176474  7262 consensus_peers.cc:485] T da11fa181227431f9119eba713d32e1a P 52b3f590e1b14b14ad7d3b827e9b21f3 -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.190872  7391 consensus_peers.cc:485] T aec8be7dfe664e29ad5153c154d4f585 P a5b21a0f245140b293e607e8f554fa1f -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.207201  7129 consensus_peers.cc:485] T 62c40420104d40e6853ea6fbbf9db21a P 136680942cc947b0b19be2f50bb114eb -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.207369  7129 consensus_peers.cc:485] T 53343926b3f14cfa9c3beb3de20eb35b P 136680942cc947b0b19be2f50bb114eb -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.207465  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 7113
W20240502 08:17:01.210680  7262 consensus_peers.cc:485] T c9282750898d4375a36d4fd0be7008ec P 52b3f590e1b14b14ad7d3b827e9b21f3 -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.232678  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 7244
I20240502 08:17:01.240070  7367 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:01.265115  7391 consensus_peers.cc:485] T 1e0ba483f04745cbb3522ece53fb2567 P a5b21a0f245140b293e607e8f554fa1f -> Peer 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791): Couldn't send request to peer 136680942cc947b0b19be2f50bb114eb. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.67:36791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.312974  7522 consensus_peers.cc:485] T dc7de5d2bd3b4518b1c4d7389f7c5fc0 P 881e59a197394322ade814528d5e2cdf -> Peer 136680942cc947b0b19be2f50bb114eb (127.6.151.67:36791): Couldn't send request to peer 136680942cc947b0b19be2f50bb114eb. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.67:36791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.351608  7391 consensus_peers.cc:485] T 1b3bec7be4924971bde466a43efaa585 P a5b21a0f245140b293e607e8f554fa1f -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.355156  7524 consensus_peers.cc:485] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf -> Peer b8432324fa494be3a41163adc25b0865 (127.6.151.66:46195): Couldn't send request to peer b8432324fa494be3a41163adc25b0865. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:46195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.387648  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 7244
W20240502 08:17:01.397667  7391 consensus_peers.cc:485] T 19c8e9eca3de4a959109f870f9299dd4 P a5b21a0f245140b293e607e8f554fa1f -> Peer cc93fe73601544fe8a8aa013f8de095b (127.6.151.65:40359): Couldn't send request to peer cc93fe73601544fe8a8aa013f8de095b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:40359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:17:01.411834  7524 connection.cc:535] server connection from 127.6.151.68:60873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:01.412391  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 7375
I20240502 08:17:01.418398  7498 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:01.435171  7525 consensus_peers.cc:485] T 05abbc25f6f6470590713c894affd936 P 881e59a197394322ade814528d5e2cdf -> Peer 52b3f590e1b14b14ad7d3b827e9b21f3 (127.6.151.68:34071): Couldn't send request to peer 52b3f590e1b14b14ad7d3b827e9b21f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.68:34071: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:01.554016  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 7375
I20240502 08:17:01.578927  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 7506
I20240502 08:17:01.584861  7629 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:01.714376  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 7506
I20240502 08:17:01.738806  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 6761
I20240502 08:17:01.739828  6822 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:01.891661  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 6761
2024-05-02T08:17:01Z chronyd exiting
[       OK ] LocationAwareRebalancingBasicTest.Basic (31513 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (31513 ms total)

[----------] 1 test from RebalanceStartCriteriaTest
[ RUN      ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240502 08:17:02.050549  6749 test_util.cc:268] Using random seed: 978141392
I20240502 08:17:02.052237  6749 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:17:02.052325  6749 ts_itest-base.cc:113] --------------
I20240502 08:17:02.052392  6749 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:17:02.052474  6749 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:17:02.052525  6749 ts_itest-base.cc:116] --------------
2024-05-02T08:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:17:02Z Disabled control of system clock
I20240502 08:17:02.088614  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:35061
--webserver_interface=127.6.151.126
--webserver_port=0
--builtin_ntp_servers=127.6.151.84:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.151.126:35061
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 08:17:02.417447  9100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:02.417798  9100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:02.417943  9100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:02.425346  9100 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:02.425513  9100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:17:02.425598  9100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:02.425683  9100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:02.425762  9100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:17:02.435446  9100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.151.126:35061
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.151.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:02.436823  9100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:02.438584  9100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:02.450167  9106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:02.450206  9108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:02.450556  9105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:02.451146  9100 server_base.cc:1034] running on GCE node
I20240502 08:17:02.451929  9100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:02.453392  9100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:02.454591  9100 hybrid_clock.cc:648] HybridClock initialized: now 1714637822454545 us; error 40 us; skew 500 ppm
I20240502 08:17:02.455005  9100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:02.457271  9100 webserver.cc:460] Webserver started at http://127.6.151.126:36909/ using document root <none> and password file <none>
I20240502 08:17:02.458006  9100 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:02.458122  9100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:02.458449  9100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:02.461125  9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3289492ddb9d44d1ac94fd246cf3a74c"
format_stamp: "Formatted at 2024-05-02 08:17:02 on dist-test-slave-x3qn"
I20240502 08:17:02.461898  9100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3289492ddb9d44d1ac94fd246cf3a74c"
format_stamp: "Formatted at 2024-05-02 08:17:02 on dist-test-slave-x3qn"
I20240502 08:17:02.468047  9100 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.004s
I20240502 08:17:02.473098  9114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:02.474951  9100 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:17:02.475121  9100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3289492ddb9d44d1ac94fd246cf3a74c"
format_stamp: "Formatted at 2024-05-02 08:17:02 on dist-test-slave-x3qn"
I20240502 08:17:02.475308  9100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:02.496222  9100 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:02.497045  9100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:02.497280  9100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:02.515236  9100 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.126:35061
I20240502 08:17:02.515255  9165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.126:35061 every 8 connection(s)
I20240502 08:17:02.516579  9100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:17:02.520401  9166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:02.521520  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 9100
I20240502 08:17:02.521708  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:17:02.534226  9166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c: Bootstrap starting.
I20240502 08:17:02.537704  9166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:02.538873  9166 log.cc:824] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:02.541260  9166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c: No bootstrap required, opened a new log
I20240502 08:17:02.547565  9166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35061 } }
I20240502 08:17:02.547926  9166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:02.548038  9166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3289492ddb9d44d1ac94fd246cf3a74c, State: Initialized, Role: FOLLOWER
I20240502 08:17:02.548647  9166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35061 } }
I20240502 08:17:02.548880  9166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:02.549017  9166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:02.549166  9166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:02.551404  9166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35061 } }
I20240502 08:17:02.551909  9166 leader_election.cc:302] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3289492ddb9d44d1ac94fd246cf3a74c; no voters: 
I20240502 08:17:02.552495  9166 leader_election.cc:288] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:17:02.552618  9171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:02.553278  9171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [term 1 LEADER]: Becoming Leader. State: Replica: 3289492ddb9d44d1ac94fd246cf3a74c, State: Running, Role: LEADER
I20240502 08:17:02.553840  9171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35061 } }
I20240502 08:17:02.554450  9166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:02.556708  9173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3289492ddb9d44d1ac94fd246cf3a74c. Latest consensus state: current_term: 1 leader_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35061 } } }
I20240502 08:17:02.557129  9173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:02.559199  9180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:02.559167  9172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3289492ddb9d44d1ac94fd246cf3a74c" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35061 } } }
I20240502 08:17:02.559618  9172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:02.564782  9180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:02.572530  9180 catalog_manager.cc:1347] Generated new cluster ID: ca78aaad91b8402fbe6713df9cc74eea
I20240502 08:17:02.572683  9180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:02.589450  9180 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:02.590701  9180 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:02.603164  9180 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3289492ddb9d44d1ac94fd246cf3a74c: Generated new TSK 0
I20240502 08:17:02.604171  9180 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:02.618757  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.65:0
--local_ip_for_outbound_sockets=127.6.151.65
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--builtin_ntp_servers=127.6.151.84:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:02.953155  9190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:02.953526  9190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:02.953636  9190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:02.961344  9190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:02.961481  9190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:02.961614  9190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.65
I20240502 08:17:02.970896  9190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.65
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:02.972312  9190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:02.974119  9190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:02.986097  9195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:02.986318  9196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:02.986768  9198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:02.987061  9190 server_base.cc:1034] running on GCE node
I20240502 08:17:02.987623  9190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:02.988554  9190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:02.989758  9190 hybrid_clock.cc:648] HybridClock initialized: now 1714637822989716 us; error 33 us; skew 500 ppm
I20240502 08:17:02.990089  9190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:02.992406  9190 webserver.cc:460] Webserver started at http://127.6.151.65:40661/ using document root <none> and password file <none>
I20240502 08:17:02.993139  9190 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:02.993244  9190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:02.993571  9190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:02.996205  9190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "eb2bb4d65bab4dd3ad1446c0acb74109"
format_stamp: "Formatted at 2024-05-02 08:17:02 on dist-test-slave-x3qn"
I20240502 08:17:02.996955  9190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "eb2bb4d65bab4dd3ad1446c0acb74109"
format_stamp: "Formatted at 2024-05-02 08:17:02 on dist-test-slave-x3qn"
I20240502 08:17:03.002171  9190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240502 08:17:03.005654  9204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.007055  9190 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:17:03.007218  9190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eb2bb4d65bab4dd3ad1446c0acb74109"
format_stamp: "Formatted at 2024-05-02 08:17:02 on dist-test-slave-x3qn"
I20240502 08:17:03.007421  9190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:03.027741  9190 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:03.028560  9190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:03.028838  9190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:03.029758  9190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:03.031224  9190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:03.031322  9190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.031435  9190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:03.031507  9190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.063786  9190 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.65:45109
I20240502 08:17:03.063840  9316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.65:45109 every 8 connection(s)
I20240502 08:17:03.065179  9190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:03.065837  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 9190
I20240502 08:17:03.066084  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:17:03.069664  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.66:0
--local_ip_for_outbound_sockets=127.6.151.66
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--builtin_ntp_servers=127.6.151.84:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:03.079109  9317 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35061
I20240502 08:17:03.079419  9317 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:03.080302  9317 heartbeater.cc:511] Master 127.6.151.126:35061 requested a full tablet report, sending...
I20240502 08:17:03.082326  9131 ts_manager.cc:194] Registered new tserver with Master: eb2bb4d65bab4dd3ad1446c0acb74109 (127.6.151.65:45109)
I20240502 08:17:03.083833  9131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.65:56143
W20240502 08:17:03.403002  9321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:03.403316  9321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:03.403474  9321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:03.410847  9321 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:03.411000  9321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:03.411181  9321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.66
I20240502 08:17:03.420704  9321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.66
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:03.422031  9321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:03.423769  9321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:03.435441  9326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:03.435400  9327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:03.436118  9329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:03.436913  9321 server_base.cc:1034] running on GCE node
I20240502 08:17:03.437572  9321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:03.438495  9321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:03.439713  9321 hybrid_clock.cc:648] HybridClock initialized: now 1714637823439650 us; error 54 us; skew 500 ppm
I20240502 08:17:03.440021  9321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:03.442173  9321 webserver.cc:460] Webserver started at http://127.6.151.66:46717/ using document root <none> and password file <none>
I20240502 08:17:03.442996  9321 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:03.443125  9321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:03.443444  9321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:03.445969  9321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6f2aec04f8c54d8d998357028b667247"
format_stamp: "Formatted at 2024-05-02 08:17:03 on dist-test-slave-x3qn"
I20240502 08:17:03.446743  9321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6f2aec04f8c54d8d998357028b667247"
format_stamp: "Formatted at 2024-05-02 08:17:03 on dist-test-slave-x3qn"
I20240502 08:17:03.451803  9321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:17:03.455138  9335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.456423  9321 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 08:17:03.456588  9321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6f2aec04f8c54d8d998357028b667247"
format_stamp: "Formatted at 2024-05-02 08:17:03 on dist-test-slave-x3qn"
I20240502 08:17:03.456779  9321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:03.473683  9321 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:03.474525  9321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:03.474814  9321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:03.475751  9321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:03.477205  9321 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:03.477303  9321 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.477420  9321 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:03.477520  9321 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.510104  9321 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.66:38373
I20240502 08:17:03.510152  9447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.66:38373 every 8 connection(s)
I20240502 08:17:03.511492  9321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:17:03.514523  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 9321
I20240502 08:17:03.514811  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:17:03.518365  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:0
--local_ip_for_outbound_sockets=127.6.151.67
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--builtin_ntp_servers=127.6.151.84:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:03.526355  9448 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35061
I20240502 08:17:03.526696  9448 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:03.527479  9448 heartbeater.cc:511] Master 127.6.151.126:35061 requested a full tablet report, sending...
I20240502 08:17:03.530021  9131 ts_manager.cc:194] Registered new tserver with Master: 6f2aec04f8c54d8d998357028b667247 (127.6.151.66:38373)
I20240502 08:17:03.531210  9131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.66:36141
W20240502 08:17:03.845480  9451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:03.845809  9451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:03.845969  9451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:03.853626  9451 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:03.853783  9451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:03.853945  9451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:17:03.863238  9451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:03.864578  9451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:03.866389  9451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:03.878851  9456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:03.878945  9457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:03.879276  9459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:03.879624  9451 server_base.cc:1034] running on GCE node
I20240502 08:17:03.880338  9451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:03.881294  9451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:03.882529  9451 hybrid_clock.cc:648] HybridClock initialized: now 1714637823882480 us; error 30 us; skew 500 ppm
I20240502 08:17:03.882885  9451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:03.885180  9451 webserver.cc:460] Webserver started at http://127.6.151.67:37119/ using document root <none> and password file <none>
I20240502 08:17:03.885876  9451 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:03.885980  9451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:03.886258  9451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:03.888819  9451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ddf7e986e6e64121bc3aac1872187477"
format_stamp: "Formatted at 2024-05-02 08:17:03 on dist-test-slave-x3qn"
I20240502 08:17:03.889536  9451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ddf7e986e6e64121bc3aac1872187477"
format_stamp: "Formatted at 2024-05-02 08:17:03 on dist-test-slave-x3qn"
I20240502 08:17:03.894591  9451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20240502 08:17:03.897931  9465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.899389  9451 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 08:17:03.899555  9451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ddf7e986e6e64121bc3aac1872187477"
format_stamp: "Formatted at 2024-05-02 08:17:03 on dist-test-slave-x3qn"
I20240502 08:17:03.899749  9451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:03.924327  9451 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:03.925208  9451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:03.925487  9451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:03.926491  9451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:03.928056  9451 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:03.928153  9451 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.928268  9451 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:03.928346  9451 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:03.960598  9451 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:46579
I20240502 08:17:03.960671  9577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:46579 every 8 connection(s)
I20240502 08:17:03.961930  9451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:17:03.965883  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 9451
I20240502 08:17:03.966156  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:17:03.969313  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:0
--local_ip_for_outbound_sockets=127.6.151.68
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--builtin_ntp_servers=127.6.151.84:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:03.974915  9578 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35061
I20240502 08:17:03.975216  9578 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:03.976023  9578 heartbeater.cc:511] Master 127.6.151.126:35061 requested a full tablet report, sending...
I20240502 08:17:03.977453  9131 ts_manager.cc:194] Registered new tserver with Master: ddf7e986e6e64121bc3aac1872187477 (127.6.151.67:46579)
I20240502 08:17:03.978255  9131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:51181
I20240502 08:17:04.086535  9317 heartbeater.cc:503] Master 127.6.151.126:35061 was elected leader, sending a full tablet report...
W20240502 08:17:04.305203  9581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:04.305500  9581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:04.305588  9581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:04.313027  9581 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:04.313154  9581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:04.313277  9581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:17:04.322369  9581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:04.323693  9581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:04.325349  9581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:04.337126  9587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:04.337353  9589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:04.337723  9586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:04.338050  9581 server_base.cc:1034] running on GCE node
I20240502 08:17:04.338629  9581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:04.339560  9581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:04.340761  9581 hybrid_clock.cc:648] HybridClock initialized: now 1714637824340705 us; error 47 us; skew 500 ppm
I20240502 08:17:04.341045  9581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:04.343384  9581 webserver.cc:460] Webserver started at http://127.6.151.68:39503/ using document root <none> and password file <none>
I20240502 08:17:04.344089  9581 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:04.344182  9581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:04.344455  9581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:04.346988  9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a520ba7b7b834ad8b63258f8e59567aa"
format_stamp: "Formatted at 2024-05-02 08:17:04 on dist-test-slave-x3qn"
I20240502 08:17:04.347710  9581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a520ba7b7b834ad8b63258f8e59567aa"
format_stamp: "Formatted at 2024-05-02 08:17:04 on dist-test-slave-x3qn"
I20240502 08:17:04.352880  9581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
I20240502 08:17:04.356415  9595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:04.357808  9581 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:17:04.357964  9581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a520ba7b7b834ad8b63258f8e59567aa"
format_stamp: "Formatted at 2024-05-02 08:17:04 on dist-test-slave-x3qn"
I20240502 08:17:04.358160  9581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:04.400204  9581 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:04.401091  9581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:04.401361  9581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:04.402307  9581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:04.404148  9581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:04.404255  9581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:04.404384  9581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:04.404461  9581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:04.435140  9581 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:43437
I20240502 08:17:04.435256  9707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:43437 every 8 connection(s)
I20240502 08:17:04.436565  9581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:04.443936  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 9581
I20240502 08:17:04.444109  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:17:04.447491  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:0
--local_ip_for_outbound_sockets=127.6.151.69
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--builtin_ntp_servers=127.6.151.84:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:04.449054  9708 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35061
I20240502 08:17:04.449376  9708 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:04.450423  9708 heartbeater.cc:511] Master 127.6.151.126:35061 requested a full tablet report, sending...
I20240502 08:17:04.452060  9131 ts_manager.cc:194] Registered new tserver with Master: a520ba7b7b834ad8b63258f8e59567aa (127.6.151.68:43437)
I20240502 08:17:04.453044  9131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:36195
I20240502 08:17:04.534263  9448 heartbeater.cc:503] Master 127.6.151.126:35061 was elected leader, sending a full tablet report...
W20240502 08:17:04.777491  9711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:04.777797  9711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:04.777889  9711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:04.785337  9711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:04.785470  9711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:04.785601  9711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:17:04.794517  9711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35061
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:04.795802  9711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:04.797425  9711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:04.809301  9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:04.809567  9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:04.809746  9719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:04.810083  9711 server_base.cc:1034] running on GCE node
I20240502 08:17:04.810732  9711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:04.811687  9711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:04.812901  9711 hybrid_clock.cc:648] HybridClock initialized: now 1714637824812854 us; error 34 us; skew 500 ppm
I20240502 08:17:04.813203  9711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:04.815469  9711 webserver.cc:460] Webserver started at http://127.6.151.69:33381/ using document root <none> and password file <none>
I20240502 08:17:04.816202  9711 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:04.816320  9711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:04.816651  9711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:04.819283  9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "940c4579995e4508b556c54e9f037cfa"
format_stamp: "Formatted at 2024-05-02 08:17:04 on dist-test-slave-x3qn"
I20240502 08:17:04.820041  9711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "940c4579995e4508b556c54e9f037cfa"
format_stamp: "Formatted at 2024-05-02 08:17:04 on dist-test-slave-x3qn"
I20240502 08:17:04.825122  9711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.004s
I20240502 08:17:04.828548  9725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:04.829919  9711 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:17:04.830099  9711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "940c4579995e4508b556c54e9f037cfa"
format_stamp: "Formatted at 2024-05-02 08:17:04 on dist-test-slave-x3qn"
I20240502 08:17:04.830307  9711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:04.848721  9711 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:04.849617  9711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:04.849929  9711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:04.850961  9711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:04.852494  9711 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:04.852581  9711 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:04.852663  9711 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:04.852718  9711 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:04.885002  9711 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:42699
I20240502 08:17:04.885052  9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:42699 every 8 connection(s)
I20240502 08:17:04.886303  9711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:04.892287  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 9711
I20240502 08:17:04.892479  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:17:04.905520  9838 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35061
I20240502 08:17:04.905867  9838 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:04.906942  9838 heartbeater.cc:511] Master 127.6.151.126:35061 requested a full tablet report, sending...
I20240502 08:17:04.908423  9131 ts_manager.cc:194] Registered new tserver with Master: 940c4579995e4508b556c54e9f037cfa (127.6.151.69:42699)
I20240502 08:17:04.909197  9131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:51199
I20240502 08:17:04.920924  6749 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:17:04.936657  9131 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54738:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 08:17:04.979012  9252 tablet_service.cc:1467] Processing CreateTablet for tablet b753309892b442a5905196e640e95319 (DEFAULT_TABLE table=TestTable [id=27a469e3e7674d9d8e4a6f51d47b8692]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.980661  9252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b753309892b442a5905196e640e95319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.980865  9578 heartbeater.cc:503] Master 127.6.151.126:35061 was elected leader, sending a full tablet report...
I20240502 08:17:04.981120  9773 tablet_service.cc:1467] Processing CreateTablet for tablet b753309892b442a5905196e640e95319 (DEFAULT_TABLE table=TestTable [id=27a469e3e7674d9d8e4a6f51d47b8692]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.982851  9773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b753309892b442a5905196e640e95319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.982426  9643 tablet_service.cc:1467] Processing CreateTablet for tablet b753309892b442a5905196e640e95319 (DEFAULT_TABLE table=TestTable [id=27a469e3e7674d9d8e4a6f51d47b8692]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:04.983682  9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b753309892b442a5905196e640e95319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:04.993818  9856 tablet_bootstrap.cc:492] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa: Bootstrap starting.
I20240502 08:17:04.995796  9858 tablet_bootstrap.cc:492] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa: Bootstrap starting.
I20240502 08:17:04.997836  9856 tablet_bootstrap.cc:654] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.999671  9858 tablet_bootstrap.cc:654] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:04.999986  9856 log.cc:824] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:05.000551  9857 tablet_bootstrap.cc:492] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109: Bootstrap starting.
I20240502 08:17:05.000908  9858 log.cc:824] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:05.003585  9858 tablet_bootstrap.cc:492] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa: No bootstrap required, opened a new log
I20240502 08:17:05.003633  9856 tablet_bootstrap.cc:492] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa: No bootstrap required, opened a new log
I20240502 08:17:05.003911  9856 ts_tablet_manager.cc:1367] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.003s
I20240502 08:17:05.003927  9858 ts_tablet_manager.cc:1367] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa: Time spent bootstrapping tablet: real 0.008s	user 0.001s	sys 0.007s
I20240502 08:17:05.004526  9857 tablet_bootstrap.cc:654] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:05.005652  9857 log.cc:824] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:05.008600  9857 tablet_bootstrap.cc:492] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109: No bootstrap required, opened a new log
I20240502 08:17:05.008857  9857 ts_tablet_manager.cc:1367] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.006s
I20240502 08:17:05.013006  9858 raft_consensus.cc:348] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.013486  9858 raft_consensus.cc:374] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.013623  9858 raft_consensus.cc:729] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940c4579995e4508b556c54e9f037cfa, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.013815  9856 raft_consensus.cc:348] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.014377  9858 consensus_queue.cc:260] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.015601  9838 heartbeater.cc:503] Master 127.6.151.126:35061 was elected leader, sending a full tablet report...
I20240502 08:17:05.015825  9856 raft_consensus.cc:374] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.015691  9857 raft_consensus.cc:348] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.015959  9856 raft_consensus.cc:729] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a520ba7b7b834ad8b63258f8e59567aa, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.016081  9857 raft_consensus.cc:374] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:05.016182  9857 raft_consensus.cc:729] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2bb4d65bab4dd3ad1446c0acb74109, State: Initialized, Role: FOLLOWER
I20240502 08:17:05.016276  9858 ts_tablet_manager.cc:1398] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.001s
I20240502 08:17:05.016712  9856 consensus_queue.cc:260] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.016916  9857 consensus_queue.cc:260] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.018591  9857 ts_tablet_manager.cc:1398] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.002s
I20240502 08:17:05.019758  9708 heartbeater.cc:503] Master 127.6.151.126:35061 was elected leader, sending a full tablet report...
I20240502 08:17:05.020310  9856 ts_tablet_manager.cc:1398] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.004s
W20240502 08:17:05.068861  9318 tablet.cc:2369] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:05.069818  9862 raft_consensus.cc:482] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:05.070072  9862 raft_consensus.cc:504] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.072132  9862 leader_election.cc:288] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb2bb4d65bab4dd3ad1446c0acb74109 (127.6.151.65:45109), a520ba7b7b834ad8b63258f8e59567aa (127.6.151.68:43437)
I20240502 08:17:05.079368  9272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b753309892b442a5905196e640e95319" candidate_uuid: "940c4579995e4508b556c54e9f037cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" is_pre_election: true
I20240502 08:17:05.079893  9272 raft_consensus.cc:2454] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 940c4579995e4508b556c54e9f037cfa in term 0.
I20240502 08:17:05.079968  9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b753309892b442a5905196e640e95319" candidate_uuid: "940c4579995e4508b556c54e9f037cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a520ba7b7b834ad8b63258f8e59567aa" is_pre_election: true
I20240502 08:17:05.080405  9663 raft_consensus.cc:2454] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 940c4579995e4508b556c54e9f037cfa in term 0.
I20240502 08:17:05.080515  9729 leader_election.cc:302] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 940c4579995e4508b556c54e9f037cfa, eb2bb4d65bab4dd3ad1446c0acb74109; no voters: 
I20240502 08:17:05.080921  9862 raft_consensus.cc:2781] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:05.081056  9862 raft_consensus.cc:482] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:05.081171  9862 raft_consensus.cc:3037] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.083225  9862 raft_consensus.cc:504] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.083782  9862 leader_election.cc:288] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [CANDIDATE]: Term 1 election: Requested vote from peers eb2bb4d65bab4dd3ad1446c0acb74109 (127.6.151.65:45109), a520ba7b7b834ad8b63258f8e59567aa (127.6.151.68:43437)
I20240502 08:17:05.084179  9272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b753309892b442a5905196e640e95319" candidate_uuid: "940c4579995e4508b556c54e9f037cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109"
I20240502 08:17:05.084267  9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b753309892b442a5905196e640e95319" candidate_uuid: "940c4579995e4508b556c54e9f037cfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a520ba7b7b834ad8b63258f8e59567aa"
I20240502 08:17:05.084349  9272 raft_consensus.cc:3037] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.084463  9663 raft_consensus.cc:3037] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:05.086405  9272 raft_consensus.cc:2454] T b753309892b442a5905196e640e95319 P eb2bb4d65bab4dd3ad1446c0acb74109 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 940c4579995e4508b556c54e9f037cfa in term 1.
I20240502 08:17:05.086474  9663 raft_consensus.cc:2454] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 940c4579995e4508b556c54e9f037cfa in term 1.
I20240502 08:17:05.086812  9729 leader_election.cc:302] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 940c4579995e4508b556c54e9f037cfa, eb2bb4d65bab4dd3ad1446c0acb74109; no voters: 
I20240502 08:17:05.087087  9862 raft_consensus.cc:2781] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:05.087464  9862 raft_consensus.cc:686] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [term 1 LEADER]: Becoming Leader. State: Replica: 940c4579995e4508b556c54e9f037cfa, State: Running, Role: LEADER
I20240502 08:17:05.087921  9862 consensus_queue.cc:237] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } }
I20240502 08:17:05.092770  9129 catalog_manager.cc:5484] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 940c4579995e4508b556c54e9f037cfa (127.6.151.69). New cstate: current_term: 1 leader_uuid: "940c4579995e4508b556c54e9f037cfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "940c4579995e4508b556c54e9f037cfa" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 42699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:05.105273  6749 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:17:05.107357  6749 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eb2bb4d65bab4dd3ad1446c0acb74109 to finish bootstrapping
I20240502 08:17:05.116096  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6f2aec04f8c54d8d998357028b667247 to finish bootstrapping
I20240502 08:17:05.125067  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ddf7e986e6e64121bc3aac1872187477 to finish bootstrapping
I20240502 08:17:05.133663  6749 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a520ba7b7b834ad8b63258f8e59567aa to finish bootstrapping
W20240502 08:17:05.137892  9839 tablet.cc:2369] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:05.141709  6749 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 940c4579995e4508b556c54e9f037cfa to finish bootstrapping
W20240502 08:17:05.188597  9709 tablet.cc:2369] T b753309892b442a5905196e640e95319 P a520ba7b7b834ad8b63258f8e59567aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:05.586714  9862 consensus_queue.cc:1035] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a520ba7b7b834ad8b63258f8e59567aa" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.591195  9862 consensus_queue.cc:1035] T b753309892b442a5905196e640e95319 P 940c4579995e4508b556c54e9f037cfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2bb4d65bab4dd3ad1446c0acb74109" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:05.633809  9383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:05.637984  9773 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:05.641949  9513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:05.654552  9252 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:05.655649  9643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:05.813812  9383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:05.823838  9773 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:05.829993  9513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:05.833034  9643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:05.835345  9252 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.442857  9383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.453488  9773 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:06.471289  9513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.491189  9643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.504201  9252 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.678791  9383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.679539  9773 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:06.692881  9643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.693163  9513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.706472  9252 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.859553  9383 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.869573  9643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.872428  9513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:06.874887  9773 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:06.876565  9252 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:07.044615  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 9190
I20240502 08:17:07.046535  9312 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:07.140861  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 9190
I20240502 08:17:07.160070  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 9321
I20240502 08:17:07.161278  9442 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:07.254379  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 9321
I20240502 08:17:07.273111  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 9451
I20240502 08:17:07.274179  9573 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:07.365515  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 9451
I20240502 08:17:07.383523  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 9581
I20240502 08:17:07.384483  9702 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:07.485541  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 9581
I20240502 08:17:07.504783  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 9711
I20240502 08:17:07.505892  9832 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:07.600859  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 9711
I20240502 08:17:07.618559  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 9100
I20240502 08:17:07.619688  9161 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:07.732360  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 9100
2024-05-02T08:17:07Z chronyd exiting
[       OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5724 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5724 ms total)

[----------] 1 test from RebalanceParamTest
[ RUN      ] RebalanceParamTest.Rebalance/3
I20240502 08:17:07.775174  6749 test_util.cc:268] Using random seed: 983866031
I20240502 08:17:07.777103  6749 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:17:07.777179  6749 ts_itest-base.cc:113] --------------
I20240502 08:17:07.777225  6749 ts_itest-base.cc:114] 7 tablet servers
I20240502 08:17:07.777259  6749 ts_itest-base.cc:115] 2 replicas per TS
I20240502 08:17:07.777312  6749 ts_itest-base.cc:116] --------------
2024-05-02T08:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:17:07Z Disabled control of system clock
I20240502 08:17:07.810923  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:36085
--webserver_interface=127.6.151.126
--webserver_port=0
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.151.126:36085
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 08:17:08.134220 10066 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:17:08.134586 10066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:08.134713 10066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:08.134821 10066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:08.142261 10066 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:08.142410 10066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:17:08.142511 10066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:08.142577 10066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:08.142691 10066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:17:08.151891 10066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.151.126:36085
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:36085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.151.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:08.153224 10066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:08.154975 10066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:08.165740 10071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:08.165979 10072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:08.166302 10074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:08.166530 10066 server_base.cc:1034] running on GCE node
I20240502 08:17:08.167337 10066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:08.168730 10066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:08.169946 10066 hybrid_clock.cc:648] HybridClock initialized: now 1714637828169881 us; error 36 us; skew 500 ppm
I20240502 08:17:08.170285 10066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:08.172549 10066 webserver.cc:460] Webserver started at http://127.6.151.126:39263/ using document root <none> and password file <none>
I20240502 08:17:08.173255 10066 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:08.173380 10066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:08.173676 10066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:08.176488 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "108c4a65ead24022acd2eaae8cc9aece"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-x3qn"
I20240502 08:17:08.177356 10066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "108c4a65ead24022acd2eaae8cc9aece"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-x3qn"
I20240502 08:17:08.182612 10066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 08:17:08.185993 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:08.187304 10066 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:17:08.187469 10066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
uuid: "108c4a65ead24022acd2eaae8cc9aece"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-x3qn"
I20240502 08:17:08.187641 10066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:08.202921 10066 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:08.203769 10066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:08.204056 10066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:08.220634 10066 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.126:36085
I20240502 08:17:08.220669 10131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.126:36085 every 8 connection(s)
I20240502 08:17:08.221946 10066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:17:08.222442  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10066
I20240502 08:17:08.222668  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:17:08.226228 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:08.236392 10132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece: Bootstrap starting.
I20240502 08:17:08.239351 10132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:08.240283 10132 log.cc:824] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:08.242713 10132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece: No bootstrap required, opened a new log
I20240502 08:17:08.248517 10132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108c4a65ead24022acd2eaae8cc9aece" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 36085 } }
I20240502 08:17:08.248871 10132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:08.248965 10132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108c4a65ead24022acd2eaae8cc9aece, State: Initialized, Role: FOLLOWER
I20240502 08:17:08.249532 10132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108c4a65ead24022acd2eaae8cc9aece" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 36085 } }
I20240502 08:17:08.249760 10132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:08.249912 10132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:08.250072 10132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:08.252172 10132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108c4a65ead24022acd2eaae8cc9aece" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 36085 } }
I20240502 08:17:08.252606 10132 leader_election.cc:302] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 108c4a65ead24022acd2eaae8cc9aece; no voters: 
I20240502 08:17:08.253170 10132 leader_election.cc:288] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:17:08.253306 10137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:08.253926 10137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [term 1 LEADER]: Becoming Leader. State: Replica: 108c4a65ead24022acd2eaae8cc9aece, State: Running, Role: LEADER
I20240502 08:17:08.254441 10137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108c4a65ead24022acd2eaae8cc9aece" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 36085 } }
I20240502 08:17:08.255120 10132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:08.257104 10139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [sys.catalog]: SysCatalogTable state changed. Reason: New leader 108c4a65ead24022acd2eaae8cc9aece. Latest consensus state: current_term: 1 leader_uuid: "108c4a65ead24022acd2eaae8cc9aece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108c4a65ead24022acd2eaae8cc9aece" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 36085 } } }
I20240502 08:17:08.257505 10139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:08.258271 10138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "108c4a65ead24022acd2eaae8cc9aece" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108c4a65ead24022acd2eaae8cc9aece" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 36085 } } }
I20240502 08:17:08.259097 10146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:08.259202 10138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:08.264813 10146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:08.273677 10146 catalog_manager.cc:1347] Generated new cluster ID: 884a55d2717d415c8f9718075eb9dd68
I20240502 08:17:08.273816 10146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:08.292305 10146 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:08.293637 10146 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:08.312044 10146 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 108c4a65ead24022acd2eaae8cc9aece: Generated new TSK 0
I20240502 08:17:08.313221 10146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:08.321449  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.65:0
--local_ip_for_outbound_sockets=127.6.151.65
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:08.655836 10156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:08.656155 10156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:08.656252 10156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:08.664019 10156 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:08.664153 10156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:08.664289 10156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.65
I20240502 08:17:08.673488 10156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.65
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:08.674823 10156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:08.676708 10156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:08.688581 10161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:08.688762 10162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:08.689160 10164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:08.689447 10156 server_base.cc:1034] running on GCE node
I20240502 08:17:08.690017 10156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:08.690968 10156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:08.692189 10156 hybrid_clock.cc:648] HybridClock initialized: now 1714637828692123 us; error 65 us; skew 500 ppm
I20240502 08:17:08.692524 10156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:08.694878 10156 webserver.cc:460] Webserver started at http://127.6.151.65:39833/ using document root <none> and password file <none>
I20240502 08:17:08.695657 10156 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:08.695760 10156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:08.696041 10156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:08.698704 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9279bdb860044c90ace75c2f03950e02"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-x3qn"
I20240502 08:17:08.699473 10156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9279bdb860044c90ace75c2f03950e02"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-x3qn"
I20240502 08:17:08.704671 10156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:17:08.708050 10170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:08.709388 10156 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:17:08.709542 10156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9279bdb860044c90ace75c2f03950e02"
format_stamp: "Formatted at 2024-05-02 08:17:08 on dist-test-slave-x3qn"
I20240502 08:17:08.709736 10156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:08.734187 10156 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:08.735179 10156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:08.735455 10156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:08.736388 10156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:08.737876 10156 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:08.737982 10156 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:08.738103 10156 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:08.738175 10156 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:08.769917 10156 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.65:43271
I20240502 08:17:08.769961 10282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.65:43271 every 8 connection(s)
I20240502 08:17:08.771261 10156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:08.774390  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10156
I20240502 08:17:08.774591  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:17:08.778124  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.66:0
--local_ip_for_outbound_sockets=127.6.151.66
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:08.785251 10283 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:08.785539 10283 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:08.786424 10283 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:08.788558 10097 ts_manager.cc:194] Registered new tserver with Master: 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:08.790017 10097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.65:37397
W20240502 08:17:09.105019 10287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:09.105324 10287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:09.105446 10287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:09.112967 10287 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:09.113108 10287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:09.113276 10287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.66
I20240502 08:17:09.122321 10287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.66
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:09.123670 10287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:09.125358 10287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:09.137467 10292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:09.137874 10293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:09.137969 10295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:09.138310 10287 server_base.cc:1034] running on GCE node
I20240502 08:17:09.138913 10287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:09.139797 10287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:09.141007 10287 hybrid_clock.cc:648] HybridClock initialized: now 1714637829140959 us; error 41 us; skew 500 ppm
I20240502 08:17:09.141332 10287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:09.143563 10287 webserver.cc:460] Webserver started at http://127.6.151.66:35785/ using document root <none> and password file <none>
I20240502 08:17:09.144304 10287 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:09.144426 10287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:09.144740 10287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:09.147433 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f577199118d5411b98b759d6c430daae"
format_stamp: "Formatted at 2024-05-02 08:17:09 on dist-test-slave-x3qn"
I20240502 08:17:09.148188 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f577199118d5411b98b759d6c430daae"
format_stamp: "Formatted at 2024-05-02 08:17:09 on dist-test-slave-x3qn"
I20240502 08:17:09.153242 10287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:17:09.156562 10301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:09.157884 10287 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:17:09.158031 10287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f577199118d5411b98b759d6c430daae"
format_stamp: "Formatted at 2024-05-02 08:17:09 on dist-test-slave-x3qn"
I20240502 08:17:09.158242 10287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:09.175968 10287 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:09.176858 10287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:09.177120 10287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:09.178093 10287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:09.179656 10287 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:09.179752 10287 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:09.179862 10287 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:09.179934 10287 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:09.210476 10287 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.66:41685
I20240502 08:17:09.210526 10413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.66:41685 every 8 connection(s)
I20240502 08:17:09.211736 10287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:17:09.212723  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10287
I20240502 08:17:09.212924  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:17:09.216393  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:0
--local_ip_for_outbound_sockets=127.6.151.67
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:09.227758 10414 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:09.228121 10414 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:09.229205 10414 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:09.235105 10097 ts_manager.cc:194] Registered new tserver with Master: f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:09.236361 10097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.66:60845
W20240502 08:17:09.552908 10417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:09.553200 10417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:09.553326 10417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:09.560869 10417 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:09.561014 10417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:09.561163 10417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:17:09.570187 10417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:09.571555 10417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:09.573254 10417 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:09.585287 10423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:09.585448 10425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:09.586024 10422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:09.588346 10417 server_base.cc:1034] running on GCE node
I20240502 08:17:09.588995 10417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:09.589953 10417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:09.591148 10417 hybrid_clock.cc:648] HybridClock initialized: now 1714637829591099 us; error 42 us; skew 500 ppm
I20240502 08:17:09.591445 10417 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:09.593851 10417 webserver.cc:460] Webserver started at http://127.6.151.67:37143/ using document root <none> and password file <none>
I20240502 08:17:09.594576 10417 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:09.594710 10417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:09.595007 10417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:09.597594 10417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "750dbb32b0114697b81aeb7fca556e7f"
format_stamp: "Formatted at 2024-05-02 08:17:09 on dist-test-slave-x3qn"
I20240502 08:17:09.598330 10417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "750dbb32b0114697b81aeb7fca556e7f"
format_stamp: "Formatted at 2024-05-02 08:17:09 on dist-test-slave-x3qn"
I20240502 08:17:09.603502 10417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240502 08:17:09.606845 10431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:09.608112 10417 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:17:09.608264 10417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "750dbb32b0114697b81aeb7fca556e7f"
format_stamp: "Formatted at 2024-05-02 08:17:09 on dist-test-slave-x3qn"
I20240502 08:17:09.608454 10417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:09.639389 10417 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:09.640249 10417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:09.640545 10417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:09.641619 10417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:09.643096 10417 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:09.643191 10417 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:09.643303 10417 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:09.643373 10417 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:09.676771 10417 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:38465
I20240502 08:17:09.676836 10543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:38465 every 8 connection(s)
I20240502 08:17:09.678171 10417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:17:09.684284  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10417
I20240502 08:17:09.684443  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:17:09.687338  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:0
--local_ip_for_outbound_sockets=127.6.151.68
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:09.690436 10544 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:09.690740 10544 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:09.691519 10544 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:09.693022 10097 ts_manager.cc:194] Registered new tserver with Master: 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:09.694154 10097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:34757
I20240502 08:17:09.793386 10283 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
W20240502 08:17:10.012396 10547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:10.012737 10547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:10.012840 10547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:10.020262 10547 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:10.020385 10547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:10.020507 10547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:17:10.029399 10547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:10.030759 10547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:10.032461 10547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:10.044263 10553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:10.044464 10555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:10.044876 10552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:10.045315 10547 server_base.cc:1034] running on GCE node
I20240502 08:17:10.045908 10547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:10.046869 10547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:10.048063 10547 hybrid_clock.cc:648] HybridClock initialized: now 1714637830048015 us; error 44 us; skew 500 ppm
I20240502 08:17:10.048363 10547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:10.050623 10547 webserver.cc:460] Webserver started at http://127.6.151.68:43277/ using document root <none> and password file <none>
I20240502 08:17:10.051410 10547 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:10.051529 10547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:10.051841 10547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:10.054387 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2fac43dcafe6465b941acc57014c43d3"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.055182 10547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2fac43dcafe6465b941acc57014c43d3"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.060408 10547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:17:10.063652 10561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:10.064906 10547 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:17:10.065055 10547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2fac43dcafe6465b941acc57014c43d3"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.065240 10547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:10.094166 10547 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:10.095069 10547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:10.095341 10547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:10.096290 10547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:10.097784 10547 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:10.097883 10547 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:10.097995 10547 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:10.098065 10547 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:10.128947 10547 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:37265
I20240502 08:17:10.129034 10673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:37265 every 8 connection(s)
I20240502 08:17:10.130772 10547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:10.131717  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10547
I20240502 08:17:10.131866  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:17:10.135178  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:0
--local_ip_for_outbound_sockets=127.6.151.69
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:10.144791 10674 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:10.145080 10674 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:10.145844 10674 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:10.147226 10097 ts_manager.cc:194] Registered new tserver with Master: 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
I20240502 08:17:10.148414 10097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:50809
I20240502 08:17:10.240259 10414 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
W20240502 08:17:10.467507 10677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:10.467797 10677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:10.467926 10677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:10.475347 10677 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:10.475481 10677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:10.475639 10677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:17:10.484572 10677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:10.485874 10677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:10.487697 10677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:10.499349 10682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:10.499437 10685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:10.499889 10683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:10.500213 10677 server_base.cc:1034] running on GCE node
I20240502 08:17:10.500777 10677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:10.501680 10677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:10.502909 10677 hybrid_clock.cc:648] HybridClock initialized: now 1714637830502853 us; error 40 us; skew 500 ppm
I20240502 08:17:10.503260 10677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:10.505479 10677 webserver.cc:460] Webserver started at http://127.6.151.69:43663/ using document root <none> and password file <none>
I20240502 08:17:10.506209 10677 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:10.506325 10677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:10.506630 10677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:10.509264 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "54f96b9fb77540e898821b386cf6d0cb"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.510020 10677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "54f96b9fb77540e898821b386cf6d0cb"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.515034 10677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240502 08:17:10.518348 10691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:10.519640 10677 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:17:10.519793 10677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "54f96b9fb77540e898821b386cf6d0cb"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.519985 10677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:10.532716 10677 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:10.533506 10677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:10.533768 10677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:10.534689 10677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:10.536054 10677 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:10.536157 10677 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:10.536259 10677 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:10.536327 10677 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:10.567670 10677 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:44733
I20240502 08:17:10.567718 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:44733 every 8 connection(s)
I20240502 08:17:10.568914 10677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:10.578933  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10677
I20240502 08:17:10.579165  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:17:10.581408 10804 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:10.581732 10804 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:10.582633  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.70:0
--local_ip_for_outbound_sockets=127.6.151.70
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:10.582777 10804 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:10.584244 10097 ts_manager.cc:194] Registered new tserver with Master: 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733)
I20240502 08:17:10.584985 10097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:56375
I20240502 08:17:10.697127 10544 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
W20240502 08:17:10.911820 10807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:10.912112 10807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:10.912261 10807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:10.919832 10807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:10.919965 10807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:10.920150 10807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.70
I20240502 08:17:10.929430 10807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.70
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:10.930851 10807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:10.932835 10807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:10.944805 10815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:10.944943 10813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:10.945279 10812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:10.945628 10807 server_base.cc:1034] running on GCE node
I20240502 08:17:10.946177 10807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:10.947109 10807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:10.948312 10807 hybrid_clock.cc:648] HybridClock initialized: now 1714637830948265 us; error 36 us; skew 500 ppm
I20240502 08:17:10.948619 10807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:10.956879 10807 webserver.cc:460] Webserver started at http://127.6.151.70:38285/ using document root <none> and password file <none>
I20240502 08:17:10.957672 10807 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:10.957787 10807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:10.958110 10807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:10.960744 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a57c9c479b194a69b5eb8f9361819a83"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.961519 10807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a57c9c479b194a69b5eb8f9361819a83"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.966938 10807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240502 08:17:10.970392 10821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:10.971776 10807 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:17:10.971940 10807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a57c9c479b194a69b5eb8f9361819a83"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:10.972142 10807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:11.030385 10807 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:11.031265 10807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:11.031535 10807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:11.032480 10807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:11.033928 10807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:11.034027 10807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:11.034139 10807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:11.034212 10807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:11.065836 10807 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.70:33091
I20240502 08:17:11.065871 10933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.70:33091 every 8 connection(s)
I20240502 08:17:11.067270 10807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:17:11.077270  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10807
I20240502 08:17:11.077478  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:17:11.079663 10934 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:11.079972 10934 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:11.080946  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.71:0
--local_ip_for_outbound_sockets=127.6.151.71
--webserver_interface=127.6.151.71
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:11.080977 10934 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:11.082425 10097 ts_manager.cc:194] Registered new tserver with Master: a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70:33091)
I20240502 08:17:11.083530 10097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.70:58951
I20240502 08:17:11.150799 10674 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
W20240502 08:17:11.420300 10937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:11.420616 10937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:11.420754 10937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:11.428424 10937 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:11.428587 10937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:11.428756 10937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.71
I20240502 08:17:11.438206 10937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.151.71
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.71
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:11.439607 10937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:11.441326 10937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:11.453567 10942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:11.453991 10943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:11.453698 10945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:11.454478 10937 server_base.cc:1034] running on GCE node
I20240502 08:17:11.455086 10937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:11.456034 10937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:11.457233 10937 hybrid_clock.cc:648] HybridClock initialized: now 1714637831457193 us; error 52 us; skew 500 ppm
I20240502 08:17:11.457553 10937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:11.459803 10937 webserver.cc:460] Webserver started at http://127.6.151.71:36385/ using document root <none> and password file <none>
I20240502 08:17:11.460551 10937 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:11.460666 10937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:11.460968 10937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:11.463667 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "84379f2079ff403eacad196d57abd7ad"
format_stamp: "Formatted at 2024-05-02 08:17:11 on dist-test-slave-x3qn"
I20240502 08:17:11.464426 10937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "84379f2079ff403eacad196d57abd7ad"
format_stamp: "Formatted at 2024-05-02 08:17:11 on dist-test-slave-x3qn"
I20240502 08:17:11.469516 10937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:17:11.472971 10951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:11.474252 10937 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:17:11.474402 10937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "84379f2079ff403eacad196d57abd7ad"
format_stamp: "Formatted at 2024-05-02 08:17:11 on dist-test-slave-x3qn"
I20240502 08:17:11.474592 10937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:11.495127 10937 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:11.495973 10937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:11.496241 10937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:11.497191 10937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:11.498667 10937 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:11.498770 10937 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:11.498874 10937 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:11.498943 10937 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:11.530539 10937 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.71:43045
I20240502 08:17:11.530586 11063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.71:43045 every 8 connection(s)
I20240502 08:17:11.531868 10937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:17:11.535416  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 10937
I20240502 08:17:11.535640  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:17:11.544580 11064 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:11.544847 11064 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:11.545580 11064 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:11.546833 10097 ts_manager.cc:194] Registered new tserver with Master: 84379f2079ff403eacad196d57abd7ad (127.6.151.71:43045)
I20240502 08:17:11.547557 10097 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.71:57373
I20240502 08:17:11.551517  6749 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:17:11.568324 10097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45754:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 08:17:11.588074 10804 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
I20240502 08:17:11.611693 10479 tablet_service.cc:1467] Processing CreateTablet for tablet f5f8034ce87a4c85a72c53ba07f178d1 (DEFAULT_TABLE table=TestTable [id=1514492063374a1ba845bab1e8569c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:11.611693 10609 tablet_service.cc:1467] Processing CreateTablet for tablet f5f8034ce87a4c85a72c53ba07f178d1 (DEFAULT_TABLE table=TestTable [id=1514492063374a1ba845bab1e8569c19]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:11.613368 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f8034ce87a4c85a72c53ba07f178d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:11.613389 10479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f8034ce87a4c85a72c53ba07f178d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:11.627609 11080 tablet_bootstrap.cc:492] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:11.628546 11081 tablet_bootstrap.cc:492] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap starting.
I20240502 08:17:11.631984 11081 tablet_bootstrap.cc:654] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:11.631986 11080 tablet_bootstrap.cc:654] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:11.633383 11080 log.cc:824] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:11.633383 11081 log.cc:824] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:11.636198 11080 tablet_bootstrap.cc:492] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:11.636209 11081 tablet_bootstrap.cc:492] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: No bootstrap required, opened a new log
I20240502 08:17:11.636436 11081 ts_tablet_manager.cc:1367] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 08:17:11.636446 11080 ts_tablet_manager.cc:1367] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 08:17:11.642228 11080 raft_consensus.cc:348] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:11.642236 11081 raft_consensus.cc:348] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:11.642616 11080 raft_consensus.cc:374] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:11.642627 11081 raft_consensus.cc:374] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:11.642755 11081 raft_consensus.cc:729] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Initialized, Role: FOLLOWER
I20240502 08:17:11.642751 11080 raft_consensus.cc:729] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:11.643429 11080 consensus_queue.cc:260] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:11.643427 11081 consensus_queue.cc:260] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:11.645220 11080 ts_tablet_manager.cc:1398] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.005s
I20240502 08:17:11.645586 11081 ts_tablet_manager.cc:1398] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.010s
W20240502 08:17:11.681092 10545 tablet.cc:2369] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:11.740850 11084 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:11.741130 11084 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:11.742600 11084 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:11.748996 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f8034ce87a4c85a72c53ba07f178d1" candidate_uuid: "2fac43dcafe6465b941acc57014c43d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f" is_pre_election: true
I20240502 08:17:11.749444 10499 raft_consensus.cc:2454] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fac43dcafe6465b941acc57014c43d3 in term 0.
I20240502 08:17:11.750015 10564 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2fac43dcafe6465b941acc57014c43d3, 750dbb32b0114697b81aeb7fca556e7f; no voters: 
I20240502 08:17:11.750367 11084 raft_consensus.cc:2781] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:11.750512 11084 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:11.750600 11084 raft_consensus.cc:3037] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:11.752624 11084 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:11.753111 11084 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 1 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:11.753638 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f8034ce87a4c85a72c53ba07f178d1" candidate_uuid: "2fac43dcafe6465b941acc57014c43d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:11.753809 10499 raft_consensus.cc:3037] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:11.756088 10499 raft_consensus.cc:2454] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fac43dcafe6465b941acc57014c43d3 in term 1.
I20240502 08:17:11.756490 10564 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2fac43dcafe6465b941acc57014c43d3, 750dbb32b0114697b81aeb7fca556e7f; no voters: 
I20240502 08:17:11.756819 11084 raft_consensus.cc:2781] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:11.757282 11084 raft_consensus.cc:686] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEADER]: Becoming Leader. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Running, Role: LEADER
I20240502 08:17:11.757766 11084 consensus_queue.cc:237] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:11.761978 10097 catalog_manager.cc:5484] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68). New cstate: current_term: 1 leader_uuid: "2fac43dcafe6465b941acc57014c43d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:11.798790  6749 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 08:17:11.800792  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9279bdb860044c90ace75c2f03950e02 to finish bootstrapping
I20240502 08:17:11.809653  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f577199118d5411b98b759d6c430daae to finish bootstrapping
I20240502 08:17:11.818303  6749 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 750dbb32b0114697b81aeb7fca556e7f to finish bootstrapping
I20240502 08:17:11.826131  6749 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2fac43dcafe6465b941acc57014c43d3 to finish bootstrapping
I20240502 08:17:11.833096  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 54f96b9fb77540e898821b386cf6d0cb to finish bootstrapping
I20240502 08:17:11.841284  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a57c9c479b194a69b5eb8f9361819a83 to finish bootstrapping
I20240502 08:17:11.850019  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 84379f2079ff403eacad196d57abd7ad to finish bootstrapping
I20240502 08:17:11.858500  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10547
I20240502 08:17:11.859484 10669 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:11.945459 10675 tablet.cc:2369] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:11.946149  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10547
I20240502 08:17:11.963132  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10677
I20240502 08:17:11.964141 10799 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:12.042718  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10677
I20240502 08:17:12.058358  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10807
I20240502 08:17:12.059428 10929 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:12.138195 10934 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
I20240502 08:17:12.139945  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10807
I20240502 08:17:12.155838  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10937
I20240502 08:17:12.156948 11059 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:12.236793  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10937
I20240502 08:17:13.912684 11099 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:13.912904 11099 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:13.914321 11099 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
W20240502 08:17:13.915517 10432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.68:37265: connect: Connection refused (error 111)
W20240502 08:17:13.916795 10432 leader_election.cc:334] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Network error: Client connection negotiation failed: client connection to 127.6.151.68:37265: connect: Connection refused (error 111)
I20240502 08:17:13.916952 10432 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f; no voters: 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:13.917305 11099 raft_consensus.cc:2726] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:15.581631 11102 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:15.581837 11102 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:15.582356 11102 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
W20240502 08:17:15.587729 10432 leader_election.cc:334] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Network error: Client connection negotiation failed: client connection to 127.6.151.68:37265: connect: Connection refused (error 111)
I20240502 08:17:15.587869 10432 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f; no voters: 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:15.588209 11102 raft_consensus.cc:2726] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:16.004271 10096 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45754:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:17:16.025871 10349 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd1419d08a94610b725cc1ed2b04d00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:16.025910 10347 tablet_service.cc:1467] Processing CreateTablet for tablet 21631366c38a4a608cb8191903fc535d (DEFAULT_TABLE table=rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:16.025951 10348 tablet_service.cc:1467] Processing CreateTablet for tablet 75f37a6f3a254fec8b7805185d9ae80f (DEFAULT_TABLE table=rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:16.027509 10218 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd1419d08a94610b725cc1ed2b04d00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:16.028120 10349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd1419d08a94610b725cc1ed2b04d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.027570 10216 tablet_service.cc:1467] Processing CreateTablet for tablet 21631366c38a4a608cb8191903fc535d (DEFAULT_TABLE table=rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:16.028609 10348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f37a6f3a254fec8b7805185d9ae80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.028805 10217 tablet_service.cc:1467] Processing CreateTablet for tablet 75f37a6f3a254fec8b7805185d9ae80f (DEFAULT_TABLE table=rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:16.029093 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21631366c38a4a608cb8191903fc535d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.029507 10217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f37a6f3a254fec8b7805185d9ae80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.030433 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21631366c38a4a608cb8191903fc535d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.031168 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd1419d08a94610b725cc1ed2b04d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:16.045082 11114 tablet_bootstrap.cc:492] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:16.049131 11114 tablet_bootstrap.cc:654] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:16.050299 11113 tablet_bootstrap.cc:492] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:16.050550 11114 log.cc:824] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:16.054378 11113 tablet_bootstrap.cc:654] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:16.055663 11113 log.cc:824] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:16.057940 11114 tablet_bootstrap.cc:492] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:16.058229 11114 ts_tablet_manager.cc:1367] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.004s
I20240502 08:17:16.064574 11113 tablet_bootstrap.cc:492] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:16.064891 11113 ts_tablet_manager.cc:1367] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.015s	user 0.003s	sys 0.008s
I20240502 08:17:16.065572 11114 raft_consensus.cc:348] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.066061 11114 raft_consensus.cc:374] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:16.066175 11114 raft_consensus.cc:729] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:16.066959 11114 consensus_queue.cc:260] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.068792 11114 ts_tablet_manager.cc:1398] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.001s
I20240502 08:17:16.069214 11114 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:16.070973 11114 tablet_bootstrap.cc:654] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:16.073976 11114 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:16.074123 11114 ts_tablet_manager.cc:1367] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:17:16.073902 11113 raft_consensus.cc:348] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.074395 11113 raft_consensus.cc:374] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:16.074507 11113 raft_consensus.cc:729] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:16.075165 11114 raft_consensus.cc:348] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.075354 11114 raft_consensus.cc:374] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:16.075441 11114 raft_consensus.cc:729] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:16.075335 11113 consensus_queue.cc:260] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.075740 11114 consensus_queue.cc:260] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.076361 11114 ts_tablet_manager.cc:1398] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 08:17:16.076645 11114 tablet_bootstrap.cc:492] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:16.077153 11113 ts_tablet_manager.cc:1398] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.003s
I20240502 08:17:16.077593 11113 tablet_bootstrap.cc:492] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:16.078352 11114 tablet_bootstrap.cc:654] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:16.079358 11113 tablet_bootstrap.cc:654] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:16.081264 11114 tablet_bootstrap.cc:492] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:16.081413 11114 ts_tablet_manager.cc:1367] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 08:17:16.081451 11113 tablet_bootstrap.cc:492] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:16.081650 11113 ts_tablet_manager.cc:1367] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.004s
I20240502 08:17:16.082108 11114 raft_consensus.cc:348] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.082281 11114 raft_consensus.cc:374] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:16.082355 11114 raft_consensus.cc:729] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:16.082592 11114 consensus_queue.cc:260] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.082710 11113 raft_consensus.cc:348] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.082926 11113 raft_consensus.cc:374] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:16.083058 11113 raft_consensus.cc:729] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:16.083169 11114 ts_tablet_manager.cc:1398] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:17:16.083287 11113 consensus_queue.cc:260] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.083719 11113 ts_tablet_manager.cc:1398] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:17:16.084018 11113 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:16.085564 11113 tablet_bootstrap.cc:654] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:16.087819 11113 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:16.088004 11113 ts_tablet_manager.cc:1367] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 08:17:16.088522 11113 raft_consensus.cc:348] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.088680 11113 raft_consensus.cc:374] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:16.088733 11113 raft_consensus.cc:729] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:16.088901 11113 consensus_queue.cc:260] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.089332 11113 ts_tablet_manager.cc:1398] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:17:16.145663 11119 raft_consensus.cc:482] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:16.146231 11119 raft_consensus.cc:504] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.147972 11119 leader_election.cc:288] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:16.154148 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21631366c38a4a608cb8191903fc535d" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02" is_pre_election: true
I20240502 08:17:16.154628 10238 raft_consensus.cc:2454] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:16.155339 10303 leader_election.cc:302] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:16.155706 11119 raft_consensus.cc:2781] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:16.155853 11119 raft_consensus.cc:482] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:16.155941 11119 raft_consensus.cc:3037] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:16.157985 11119 raft_consensus.cc:504] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.158514 11119 leader_election.cc:288] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:16.159124 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21631366c38a4a608cb8191903fc535d" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:16.159317 10238 raft_consensus.cc:3037] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:16.161437 10238 raft_consensus.cc:2454] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:16.161836 10303 leader_election.cc:302] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:16.162140 11119 raft_consensus.cc:2781] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:16.162654 11119 raft_consensus.cc:686] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:16.163177 11119 consensus_queue.cc:237] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.167464 10096 catalog_manager.cc:5484] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:16.217625 10415 tablet.cc:2369] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:17:16.278240 10284 tablet.cc:2369] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:17:16.360375 11119 raft_consensus.cc:482] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:16.360633 11119 raft_consensus.cc:504] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.361143 11119 leader_election.cc:288] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:16.361841 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd1419d08a94610b725cc1ed2b04d00" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02" is_pre_election: true
I20240502 08:17:16.362078 10238 raft_consensus.cc:2454] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:16.362598 10303 leader_election.cc:302] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:16.362968 11119 raft_consensus.cc:2781] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:16.363049 11119 raft_consensus.cc:482] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:16.363142 11119 raft_consensus.cc:3037] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:16.365052 11119 raft_consensus.cc:504] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.365518 11119 leader_election.cc:288] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:16.366160 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd1419d08a94610b725cc1ed2b04d00" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:16.366362 10238 raft_consensus.cc:3037] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:16.368306 10238 raft_consensus.cc:2454] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:16.368711 10303 leader_election.cc:302] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:16.369010 11119 raft_consensus.cc:2781] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:16.369156 11119 raft_consensus.cc:686] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:16.369416 11119 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.371855 10096 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:16.470525 11119 raft_consensus.cc:482] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:16.470749 11119 raft_consensus.cc:504] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.471325 11119 leader_election.cc:288] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:16.471952 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f37a6f3a254fec8b7805185d9ae80f" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02" is_pre_election: true
I20240502 08:17:16.472165 10238 raft_consensus.cc:2454] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:16.472682 10303 leader_election.cc:302] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:16.472975 11119 raft_consensus.cc:2781] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:16.473106 11119 raft_consensus.cc:482] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:16.473196 11119 raft_consensus.cc:3037] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:16.475135 11119 raft_consensus.cc:504] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.475606 11119 leader_election.cc:288] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:16.476229 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75f37a6f3a254fec8b7805185d9ae80f" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:16.476416 10238 raft_consensus.cc:3037] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:16.478385 10238 raft_consensus.cc:2454] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:16.478883 10303 leader_election.cc:302] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:16.479219 11119 raft_consensus.cc:2781] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:16.479370 11119 raft_consensus.cc:686] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:16.479677 11119 consensus_queue.cc:237] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:16.482362 10096 catalog_manager.cc:5484] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:16.579622 11119 consensus_queue.cc:1035] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:16.825196 11124 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:16.930596 11131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:16.930979 11131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:16.938683 11131 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:16.960273 11119 consensus_queue.cc:1035] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 152.042 ms
  time per row: 0.076021 ms
I20240502 08:17:17.249646 10097 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45754:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:17:17.259229 10479 tablet_service.cc:1467] Processing CreateTablet for tablet befc5d75417d47d8b7790017c49fcb71 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:17.259459 10478 tablet_service.cc:1467] Processing CreateTablet for tablet 2b27b7cc14814dc48c1fe37dbbea9619 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:17.259627 10477 tablet_service.cc:1467] Processing CreateTablet for tablet 82835c215cce4ce49680d56a5bbf0208 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:17.259783 10479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befc5d75417d47d8b7790017c49fcb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:17.259927 10217 tablet_service.cc:1467] Processing CreateTablet for tablet 82835c215cce4ce49680d56a5bbf0208 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:17.260021 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b27b7cc14814dc48c1fe37dbbea9619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:17.260298 10217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82835c215cce4ce49680d56a5bbf0208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:17.260356 10477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82835c215cce4ce49680d56a5bbf0208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:17.265705 10216 tablet_service.cc:1467] Processing CreateTablet for tablet befc5d75417d47d8b7790017c49fcb71 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:17.266135 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befc5d75417d47d8b7790017c49fcb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:17.271474 10349 tablet_service.cc:1467] Processing CreateTablet for tablet 2b27b7cc14814dc48c1fe37dbbea9619 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:17.271947 10349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b27b7cc14814dc48c1fe37dbbea9619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:17.275207 11177 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:17.275768 11176 tablet_bootstrap.cc:492] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:17.277071 11177 tablet_bootstrap.cc:654] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:17.277566 11176 tablet_bootstrap.cc:654] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:17.279013 11177 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:17.279151 11177 ts_tablet_manager.cc:1367] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:17:17.279917 11177 raft_consensus.cc:348] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:17.280089 11177 raft_consensus.cc:374] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:17.280176 11177 raft_consensus.cc:729] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:17.280349 11177 consensus_queue.cc:260] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:17.280468 11176 tablet_bootstrap.cc:492] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:17.280582 11176 ts_tablet_manager.cc:1367] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 08:17:17.281109 11177 ts_tablet_manager.cc:1398] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:17.281352 11177 tablet_bootstrap.cc:492] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:17.281352 11176 raft_consensus.cc:348] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.281564 11176 raft_consensus.cc:374] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:17.281669 11176 raft_consensus.cc:729] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:17.282770 11177 tablet_bootstrap.cc:654] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:17.283057 11179 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:17.283097 11176 consensus_queue.cc:260] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.284133 11176 ts_tablet_manager.cc:1398] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:17:17.284457 11176 tablet_bootstrap.cc:492] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:17.286118 11176 tablet_bootstrap.cc:654] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:17.288018 11176 tablet_bootstrap.cc:492] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:17.288177 11176 ts_tablet_manager.cc:1367] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:17:17.288765 11176 raft_consensus.cc:348] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.288915 11179 tablet_bootstrap.cc:654] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:17.288937 11176 raft_consensus.cc:374] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:17.289022 11176 raft_consensus.cc:729] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:17.289247 11176 consensus_queue.cc:260] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.289682 11176 ts_tablet_manager.cc:1398] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:17:17.290606 11177 tablet_bootstrap.cc:492] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:17.290812 11177 ts_tablet_manager.cc:1367] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.006s
I20240502 08:17:17.291663 11177 raft_consensus.cc:348] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.291950 11177 raft_consensus.cc:374] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:17.292083 11177 raft_consensus.cc:729] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:17.292330 11177 consensus_queue.cc:260] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.293450 11177 ts_tablet_manager.cc:1398] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:17:17.293783 11177 tablet_bootstrap.cc:492] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:17.295456 11177 tablet_bootstrap.cc:654] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:17.297375 11179 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:17.297505 11177 tablet_bootstrap.cc:492] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:17.297573 11179 ts_tablet_manager.cc:1367] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.015s	user 0.001s	sys 0.008s
I20240502 08:17:17.297641 11177 ts_tablet_manager.cc:1367] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:17:17.298374 11177 raft_consensus.cc:348] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.298374 11179 raft_consensus.cc:348] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:17.298576 11177 raft_consensus.cc:374] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:17.298586 11179 raft_consensus.cc:374] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:17.298710 11179 raft_consensus.cc:729] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:17.298738 11177 raft_consensus.cc:729] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:17.299007 11177 consensus_queue.cc:260] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.298987 11179 consensus_queue.cc:260] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:17.299530 11177 ts_tablet_manager.cc:1398] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:17.299688 11179 ts_tablet_manager.cc:1398] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 08:17:17.338444 11117 raft_consensus.cc:482] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:17.338701 11117 raft_consensus.cc:504] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.340198 11117 leader_election.cc:288] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:17.347273 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befc5d75417d47d8b7790017c49fcb71" candidate_uuid: "9279bdb860044c90ace75c2f03950e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f" is_pre_election: true
I20240502 08:17:17.347574 10499 raft_consensus.cc:2454] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9279bdb860044c90ace75c2f03950e02 in term 0.
I20240502 08:17:17.348088 10173 leader_election.cc:302] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:17.348500 11117 raft_consensus.cc:2781] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:17.348620 11117 raft_consensus.cc:482] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:17.348690 11117 raft_consensus.cc:3037] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:17.350714 11117 raft_consensus.cc:504] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.351207 11117 leader_election.cc:288] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:17.351689 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befc5d75417d47d8b7790017c49fcb71" candidate_uuid: "9279bdb860044c90ace75c2f03950e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:17.351879 10499 raft_consensus.cc:3037] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:17.353950 10499 raft_consensus.cc:2454] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9279bdb860044c90ace75c2f03950e02 in term 1.
I20240502 08:17:17.354429 10173 leader_election.cc:302] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:17.354864 11117 raft_consensus.cc:2781] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:17.355383 11117 raft_consensus.cc:686] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Becoming Leader. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Running, Role: LEADER
I20240502 08:17:17.355899 11117 consensus_queue.cc:237] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.359555 10097 catalog_manager.cc:5484] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9279bdb860044c90ace75c2f03950e02 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:17.362358 11124 raft_consensus.cc:482] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:17.362614 11124 raft_consensus.cc:504] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:17.363397 11124 leader_election.cc:288] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:17.369699 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b27b7cc14814dc48c1fe37dbbea9619" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f" is_pre_election: true
I20240502 08:17:17.369925 10499 raft_consensus.cc:2454] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:17.370294 10304 leader_election.cc:302] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:17.370587 11124 raft_consensus.cc:2781] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:17.370747 11124 raft_consensus.cc:482] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:17.370855 11124 raft_consensus.cc:3037] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:17.372722 11124 raft_consensus.cc:504] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:17.373195 11124 leader_election.cc:288] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:17.373751 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b27b7cc14814dc48c1fe37dbbea9619" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:17.373942 10499 raft_consensus.cc:3037] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:17.375949 10499 raft_consensus.cc:2454] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:17.376366 10304 leader_election.cc:302] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:17.376704 11124 raft_consensus.cc:2781] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:17.376856 11124 raft_consensus.cc:686] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:17.377072 11124 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:17.380002 10097 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:17.511099 11181 raft_consensus.cc:482] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:17.511304 11181 raft_consensus.cc:504] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.512085 11181 leader_election.cc:288] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:17.518252 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82835c215cce4ce49680d56a5bbf0208" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02" is_pre_election: true
I20240502 08:17:17.518535 10238 raft_consensus.cc:2454] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 0.
I20240502 08:17:17.519052 10433 leader_election.cc:302] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:17.519357 11181 raft_consensus.cc:2781] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:17.519486 11181 raft_consensus.cc:482] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:17.519570 11181 raft_consensus.cc:3037] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:17.521580 11181 raft_consensus.cc:504] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.522079 11181 leader_election.cc:288] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:17.522694 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82835c215cce4ce49680d56a5bbf0208" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:17.522905 10238 raft_consensus.cc:3037] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:17.524828 10238 raft_consensus.cc:2454] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 1.
I20240502 08:17:17.525205 10433 leader_election.cc:302] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:17.525512 11181 raft_consensus.cc:2781] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:17.525928 11181 raft_consensus.cc:686] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Becoming Leader. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Running, Role: LEADER
I20240502 08:17:17.526427 11181 consensus_queue.cc:237] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } }
I20240502 08:17:17.531033 10097 catalog_manager.cc:5484] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67). New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:17.702127 11181 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:17.702381 11181 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:17.702984 11181 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
W20240502 08:17:17.704979 10432 leader_election.cc:334] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Network error: Client connection negotiation failed: client connection to 127.6.151.68:37265: connect: Connection refused (error 111)
I20240502 08:17:17.705147 10432 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f; no voters: 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:17.705427 11181 raft_consensus.cc:2726] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:17.827842 11124 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:17.907804 11196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:17.908113 11196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:17.915757 11196 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 08:17:17.967054 11221 consensus_queue.cc:1035] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:18.042970 11181 consensus_queue.cc:1035] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 228.083 ms
  time per row: 0.114041 ms
I20240502 08:17:18.290572 10094 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45754:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:17:18.300449 10349 tablet_service.cc:1467] Processing CreateTablet for tablet b02fadb91da44d33bd8f45847433a1ab (DEFAULT_TABLE table=rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:18.300514 10347 tablet_service.cc:1467] Processing CreateTablet for tablet a2c5cbee8e5546739f31b047565e1144 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:18.301019 10349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02fadb91da44d33bd8f45847433a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:18.301231 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c5cbee8e5546739f31b047565e1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:18.301270 10479 tablet_service.cc:1467] Processing CreateTablet for tablet a2c5cbee8e5546739f31b047565e1144 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:18.301455 10478 tablet_service.cc:1467] Processing CreateTablet for tablet b02fadb91da44d33bd8f45847433a1ab (DEFAULT_TABLE table=rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:18.301653 10479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c5cbee8e5546739f31b047565e1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:18.301856 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02fadb91da44d33bd8f45847433a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:18.303153 10216 tablet_service.cc:1467] Processing CreateTablet for tablet d426a029b71e4dd4a8852a5cfe708ae0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:18.303550 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d426a029b71e4dd4a8852a5cfe708ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:18.307482 10348 tablet_service.cc:1467] Processing CreateTablet for tablet d426a029b71e4dd4a8852a5cfe708ae0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:18.307987 10348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d426a029b71e4dd4a8852a5cfe708ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:18.315985 11255 tablet_bootstrap.cc:492] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:18.317902 11255 tablet_bootstrap.cc:654] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:18.318856 11256 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:18.321316 11256 tablet_bootstrap.cc:654] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:18.321301 11257 tablet_bootstrap.cc:492] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:18.323241 11255 tablet_bootstrap.cc:492] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:18.323249 11257 tablet_bootstrap.cc:654] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:18.323288 11256 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:18.323403 11255 ts_tablet_manager.cc:1367] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.006s
I20240502 08:17:18.323436 11256 ts_tablet_manager.cc:1367] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.005s	user 0.000s	sys 0.003s
I20240502 08:17:18.324031 11256 raft_consensus.cc:348] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.324312 11256 raft_consensus.cc:374] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:18.324249 11255 raft_consensus.cc:348] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.324422 11256 raft_consensus.cc:729] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:18.324438 11255 raft_consensus.cc:374] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:18.324527 11255 raft_consensus.cc:729] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:18.324743 11256 consensus_queue.cc:260] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.324780 11255 consensus_queue.cc:260] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.325219 11256 ts_tablet_manager.cc:1398] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:17:18.325536 11256 tablet_bootstrap.cc:492] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:18.326226 11255 ts_tablet_manager.cc:1398] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 08:17:18.326522 11255 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:18.326786 11257 tablet_bootstrap.cc:492] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:18.326963 11257 ts_tablet_manager.cc:1367] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 08:17:18.327298 11256 tablet_bootstrap.cc:654] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:18.328318 11255 tablet_bootstrap.cc:654] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:18.329246 11257 raft_consensus.cc:348] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:18.329450 11257 raft_consensus.cc:374] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:18.329556 11257 raft_consensus.cc:729] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:18.329828 11257 consensus_queue.cc:260] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:18.333251 11256 tablet_bootstrap.cc:492] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:18.333416 11256 ts_tablet_manager.cc:1367] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.008s	user 0.000s	sys 0.006s
I20240502 08:17:18.334003 11256 raft_consensus.cc:348] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.334179 11256 raft_consensus.cc:374] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:18.334262 11256 raft_consensus.cc:729] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:18.334431 11257 ts_tablet_manager.cc:1398] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.000s
I20240502 08:17:18.334507 11256 consensus_queue.cc:260] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.335175 11256 ts_tablet_manager.cc:1398] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:18.336423 11255 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:18.336621 11255 ts_tablet_manager.cc:1367] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.005s
I20240502 08:17:18.337252 11255 raft_consensus.cc:348] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.337416 11255 raft_consensus.cc:374] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:18.337515 11255 raft_consensus.cc:729] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:18.337741 11255 consensus_queue.cc:260] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.338178 11255 ts_tablet_manager.cc:1398] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:17:18.338424 11255 tablet_bootstrap.cc:492] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:18.340044 11255 tablet_bootstrap.cc:654] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:18.341943 11255 tablet_bootstrap.cc:492] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:18.342093 11255 ts_tablet_manager.cc:1367] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:17:18.342716 11255 raft_consensus.cc:348] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:18.342875 11255 raft_consensus.cc:374] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:18.342976 11255 raft_consensus.cc:729] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:18.343186 11255 consensus_queue.cc:260] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:18.343616 11255 ts_tablet_manager.cc:1398] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 08:17:18.444687 11124 raft_consensus.cc:482] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:18.444909 11124 raft_consensus.cc:504] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.445451 11124 leader_election.cc:288] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:18.446107 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02fadb91da44d33bd8f45847433a1ab" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f" is_pre_election: true
I20240502 08:17:18.446296 10498 raft_consensus.cc:2454] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:18.446768 10304 leader_election.cc:302] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:18.447027 11124 raft_consensus.cc:2781] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:18.447113 11124 raft_consensus.cc:482] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:18.447232 11124 raft_consensus.cc:3037] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:18.449083 11124 raft_consensus.cc:504] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.449537 11124 leader_election.cc:288] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:18.450170 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02fadb91da44d33bd8f45847433a1ab" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:18.450371 10498 raft_consensus.cc:3037] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:18.452389 10498 raft_consensus.cc:2454] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:18.452848 10304 leader_election.cc:302] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:18.453145 11124 raft_consensus.cc:2781] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:18.453279 11124 raft_consensus.cc:686] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:18.453518 11124 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.456287 10097 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:18.478864 11124 raft_consensus.cc:482] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:18.479038 11124 raft_consensus.cc:504] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.479588 11124 leader_election.cc:288] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:18.480242 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c5cbee8e5546739f31b047565e1144" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f" is_pre_election: true
I20240502 08:17:18.480491 10498 raft_consensus.cc:2454] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:18.480998 10304 leader_election.cc:302] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:18.481287 11124 raft_consensus.cc:2781] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:18.481402 11124 raft_consensus.cc:482] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:18.481503 11124 raft_consensus.cc:3037] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:18.483577 11124 raft_consensus.cc:504] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.484019 11124 leader_election.cc:288] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:18.484570 10498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c5cbee8e5546739f31b047565e1144" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:18.484746 10498 raft_consensus.cc:3037] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:18.486599 10498 raft_consensus.cc:2454] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:18.487056 10304 leader_election.cc:302] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:18.487366 11124 raft_consensus.cc:2781] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:18.487488 11124 raft_consensus.cc:686] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:18.487779 11124 consensus_queue.cc:237] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:18.490248 10097 catalog_manager.cc:5484] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:18.612573 11225 raft_consensus.cc:482] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:18.612793 11225 raft_consensus.cc:504] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:18.613504 11225 leader_election.cc:288] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:18.619499 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d426a029b71e4dd4a8852a5cfe708ae0" candidate_uuid: "9279bdb860044c90ace75c2f03950e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f577199118d5411b98b759d6c430daae" is_pre_election: true
I20240502 08:17:18.619812 10369 raft_consensus.cc:2454] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9279bdb860044c90ace75c2f03950e02 in term 0.
I20240502 08:17:18.620260 10172 leader_election.cc:302] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:18.620584 11225 raft_consensus.cc:2781] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:18.620698 11225 raft_consensus.cc:482] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:18.620769 11225 raft_consensus.cc:3037] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:18.622766 11225 raft_consensus.cc:504] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:18.623262 11225 leader_election.cc:288] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:18.623797 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d426a029b71e4dd4a8852a5cfe708ae0" candidate_uuid: "9279bdb860044c90ace75c2f03950e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:18.623973 10369 raft_consensus.cc:3037] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:18.625911 10369 raft_consensus.cc:2454] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9279bdb860044c90ace75c2f03950e02 in term 1.
I20240502 08:17:18.626323 10172 leader_election.cc:302] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:18.626681 11225 raft_consensus.cc:2781] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:18.626844 11225 raft_consensus.cc:686] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Becoming Leader. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Running, Role: LEADER
I20240502 08:17:18.627120 11225 consensus_queue.cc:237] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:18.629599 10097 catalog_manager.cc:5484] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9279bdb860044c90ace75c2f03950e02 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:18.932863 11124 consensus_queue.cc:1035] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:19.013139 11225 consensus_queue.cc:1035] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:19.023998 11269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:19.024323 11269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:19.031860 11269 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:17:19.034668 11124 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'

INSERT report
    rows total: 2000
    time total: 226.32 ms
  time per row: 0.11316 ms
I20240502 08:17:19.402112 10093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45754:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:17:19.410667 10216 tablet_service.cc:1467] Processing CreateTablet for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:19.410822 10478 tablet_service.cc:1467] Processing CreateTablet for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:19.411145 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d1cef3b67e4d58aa76c8fd23f2c099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.411271 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d1cef3b67e4d58aa76c8fd23f2c099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.411913 10347 tablet_service.cc:1467] Processing CreateTablet for tablet fb8df38d1c3340fda6024301f0798487 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:19.412315 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8df38d1c3340fda6024301f0798487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.412495 10218 tablet_service.cc:1467] Processing CreateTablet for tablet fb8df38d1c3340fda6024301f0798487 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:19.412916 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8df38d1c3340fda6024301f0798487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.419211 10217 tablet_service.cc:1467] Processing CreateTablet for tablet bde4a6ed1250456696a769d280e19ac4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:19.419764 10217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4a6ed1250456696a769d280e19ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.423866 10479 tablet_service.cc:1467] Processing CreateTablet for tablet bde4a6ed1250456696a769d280e19ac4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:19.424438 10479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4a6ed1250456696a769d280e19ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:19.426016 11322 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:19.426661 11321 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:19.427923 11322 tablet_bootstrap.cc:654] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:19.429047 11321 tablet_bootstrap.cc:654] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:19.433488 11322 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:19.433660 11322 ts_tablet_manager.cc:1367] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.003s
I20240502 08:17:19.434517 11322 raft_consensus.cc:348] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.434785 11322 raft_consensus.cc:374] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:19.434897 11322 raft_consensus.cc:729] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.435130 11322 consensus_queue.cc:260] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.435886 11322 ts_tablet_manager.cc:1398] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.003s
I20240502 08:17:19.438019 11321 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:19.438160 11321 ts_tablet_manager.cc:1367] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.004s
I20240502 08:17:19.438890 11321 raft_consensus.cc:348] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.439105 11321 raft_consensus.cc:374] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:19.439209 11321 raft_consensus.cc:729] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.439458 11321 consensus_queue.cc:260] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.440045 11321 ts_tablet_manager.cc:1398] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:17:19.440310 11321 tablet_bootstrap.cc:492] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:19.441841 11321 tablet_bootstrap.cc:654] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:19.442050 11322 tablet_bootstrap.cc:492] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:19.444725 11322 tablet_bootstrap.cc:654] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:19.446484 11322 tablet_bootstrap.cc:492] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:19.446667 11322 ts_tablet_manager.cc:1367] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.002s
I20240502 08:17:19.446944 11323 tablet_bootstrap.cc:492] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:19.447678 11322 raft_consensus.cc:348] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:19.447932 11322 raft_consensus.cc:374] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:19.448016 11322 raft_consensus.cc:729] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.448273 11322 consensus_queue.cc:260] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:19.448848 11321 tablet_bootstrap.cc:492] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:19.449033 11321 ts_tablet_manager.cc:1367] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.009s	user 0.000s	sys 0.005s
I20240502 08:17:19.449123 11323 tablet_bootstrap.cc:654] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:19.449433 11322 ts_tablet_manager.cc:1398] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:17:19.449730 11322 tablet_bootstrap.cc:492] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:19.449819 11321 raft_consensus.cc:348] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.450037 11321 raft_consensus.cc:374] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:19.450105 11321 raft_consensus.cc:729] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.450371 11321 consensus_queue.cc:260] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.451050 11321 ts_tablet_manager.cc:1398] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:17:19.452697 11322 tablet_bootstrap.cc:654] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:19.459103 11323 tablet_bootstrap.cc:492] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:19.459270 11323 ts_tablet_manager.cc:1367] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.012s	user 0.002s	sys 0.007s
I20240502 08:17:19.459888 11323 raft_consensus.cc:348] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:19.460060 11323 raft_consensus.cc:374] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:19.460165 11323 raft_consensus.cc:729] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.460400 11323 consensus_queue.cc:260] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:19.460906 11323 ts_tablet_manager.cc:1398] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:17:19.461028 11322 tablet_bootstrap.cc:492] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:19.461187 11322 ts_tablet_manager.cc:1367] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.012s	user 0.003s	sys 0.006s
I20240502 08:17:19.461777 11322 raft_consensus.cc:348] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.461938 11322 raft_consensus.cc:374] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:19.462040 11322 raft_consensus.cc:729] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:19.462284 11322 consensus_queue.cc:260] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.462868 11322 ts_tablet_manager.cc:1398] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:19.517964 11234 raft_consensus.cc:482] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:19.518170 11234 raft_consensus.cc:504] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.518817 11234 leader_election.cc:288] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:19.519445 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4a6ed1250456696a769d280e19ac4" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02" is_pre_election: true
I20240502 08:17:19.519690 10238 raft_consensus.cc:2454] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 0.
I20240502 08:17:19.520134 10433 leader_election.cc:302] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:19.520421 11234 raft_consensus.cc:2781] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:19.520530 11234 raft_consensus.cc:482] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:19.520598 11234 raft_consensus.cc:3037] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:19.522447 11234 raft_consensus.cc:504] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.522925 11234 leader_election.cc:288] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:19.523458 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4a6ed1250456696a769d280e19ac4" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:19.523653 10238 raft_consensus.cc:3037] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:19.525523 10238 raft_consensus.cc:2454] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 1.
I20240502 08:17:19.525924 10433 leader_election.cc:302] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:19.526188 11234 raft_consensus.cc:2781] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:19.526556 11234 raft_consensus.cc:686] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Becoming Leader. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Running, Role: LEADER
I20240502 08:17:19.526827 11234 consensus_queue.cc:237] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.529347 10093 catalog_manager.cc:5484] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67). New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:19.530228 11225 raft_consensus.cc:482] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:19.530357 11225 raft_consensus.cc:504] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.530889 11225 leader_election.cc:288] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:19.531461 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d1cef3b67e4d58aa76c8fd23f2c099" candidate_uuid: "9279bdb860044c90ace75c2f03950e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f" is_pre_election: true
I20240502 08:17:19.531685 10499 raft_consensus.cc:2454] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9279bdb860044c90ace75c2f03950e02 in term 0.
I20240502 08:17:19.532131 10173 leader_election.cc:302] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:19.532392 11225 raft_consensus.cc:2781] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:19.532517 11225 raft_consensus.cc:482] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:19.532621 11225 raft_consensus.cc:3037] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:19.534510 11225 raft_consensus.cc:504] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.535005 11225 leader_election.cc:288] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:19.535465 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d1cef3b67e4d58aa76c8fd23f2c099" candidate_uuid: "9279bdb860044c90ace75c2f03950e02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:19.535686 10499 raft_consensus.cc:3037] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:19.537452 10499 raft_consensus.cc:2454] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9279bdb860044c90ace75c2f03950e02 in term 1.
I20240502 08:17:19.537878 10173 leader_election.cc:302] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:19.538219 11225 raft_consensus.cc:2781] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:19.538357 11225 raft_consensus.cc:686] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Becoming Leader. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Running, Role: LEADER
I20240502 08:17:19.538877 11225 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:19.541424 10093 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9279bdb860044c90ace75c2f03950e02 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:19.760421 11311 raft_consensus.cc:482] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:19.760645 11311 raft_consensus.cc:504] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:19.761163 11311 leader_election.cc:288] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:19.761818 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8df38d1c3340fda6024301f0798487" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02" is_pre_election: true
I20240502 08:17:19.762071 10238 raft_consensus.cc:2454] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:19.762521 10303 leader_election.cc:302] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:19.762852 11311 raft_consensus.cc:2781] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:19.762974 11311 raft_consensus.cc:482] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:19.763074 11311 raft_consensus.cc:3037] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:19.765034 11311 raft_consensus.cc:504] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:19.765533 11311 leader_election.cc:288] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:19.766119 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8df38d1c3340fda6024301f0798487" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:19.766322 10238 raft_consensus.cc:3037] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:19.768229 10238 raft_consensus.cc:2454] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:19.768657 10303 leader_election.cc:302] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:19.768975 11311 raft_consensus.cc:2781] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:19.769150 11311 raft_consensus.cc:686] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:19.769408 11311 consensus_queue.cc:237] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:19.772076 10093 catalog_manager.cc:5484] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:19.913786 11234 consensus_queue.cc:1035] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:20.066967 11225 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:20.147481 11334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:20.147850 11334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:20.155503 11334 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 08:17:20.252099 11311 consensus_queue.cc:1035] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 186.301 ms
  time per row: 0.0931504 ms
I20240502 08:17:20.490478 10093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45754:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:17:20.499642 10479 tablet_service.cc:1467] Processing CreateTablet for tablet 138b1526ab364e6bafe0abfa1edb4376 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:20.499692 10347 tablet_service.cc:1467] Processing CreateTablet for tablet 138b1526ab364e6bafe0abfa1edb4376 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:20.500125 10347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138b1526ab364e6bafe0abfa1edb4376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.500124 10479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138b1526ab364e6bafe0abfa1edb4376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.503325 10478 tablet_service.cc:1467] Processing CreateTablet for tablet c0a4038683ac48449ccad5eaeeef5694 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:20.503733 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a4038683ac48449ccad5eaeeef5694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.503839 10348 tablet_service.cc:1467] Processing CreateTablet for tablet c0a4038683ac48449ccad5eaeeef5694 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:20.503835 10217 tablet_service.cc:1467] Processing CreateTablet for tablet d471c5fb3045495aa1ec0a774e6164d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:20.504168 10348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a4038683ac48449ccad5eaeeef5694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.504261 10217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d471c5fb3045495aa1ec0a774e6164d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.511109 10349 tablet_service.cc:1467] Processing CreateTablet for tablet d471c5fb3045495aa1ec0a774e6164d8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:17:20.511669 10349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d471c5fb3045495aa1ec0a774e6164d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:20.513195 11388 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:20.514971 11388 tablet_bootstrap.cc:654] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:20.520131 11388 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:20.520366 11388 ts_tablet_manager.cc:1367] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.000s
I20240502 08:17:20.520450 11391 tablet_bootstrap.cc:492] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02: Bootstrap starting.
I20240502 08:17:20.521435 11388 raft_consensus.cc:348] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.521718 11388 raft_consensus.cc:374] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:20.521831 11388 raft_consensus.cc:729] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:20.522119 11388 consensus_queue.cc:260] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.522627 11391 tablet_bootstrap.cc:654] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:20.523514 11388 ts_tablet_manager.cc:1398] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.004s
I20240502 08:17:20.523850 11388 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Bootstrap starting.
I20240502 08:17:20.526093 11388 tablet_bootstrap.cc:654] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:20.527736 11389 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:20.529043 11391 tablet_bootstrap.cc:492] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02: No bootstrap required, opened a new log
I20240502 08:17:20.529194 11391 ts_tablet_manager.cc:1367] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240502 08:17:20.529191 11389 tablet_bootstrap.cc:654] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:20.529815 11391 raft_consensus.cc:348] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.530036 11391 raft_consensus.cc:374] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:20.530191 11391 raft_consensus.cc:729] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Initialized, Role: FOLLOWER
I20240502 08:17:20.530468 11391 consensus_queue.cc:260] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.531093 11388 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: No bootstrap required, opened a new log
I20240502 08:17:20.531258 11388 ts_tablet_manager.cc:1367] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 08:17:20.531584 11391 ts_tablet_manager.cc:1398] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:20.532035 11389 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:20.532044 11388 raft_consensus.cc:348] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.532195 11389 ts_tablet_manager.cc:1367] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:17:20.532232 11388 raft_consensus.cc:374] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:20.532296 11388 raft_consensus.cc:729] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Initialized, Role: FOLLOWER
I20240502 08:17:20.532526 11388 consensus_queue.cc:260] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.533037 11388 ts_tablet_manager.cc:1398] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:17:20.532931 11389 raft_consensus.cc:348] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.533115 11389 raft_consensus.cc:374] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:20.533218 11389 raft_consensus.cc:729] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:20.533457 11389 consensus_queue.cc:260] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.534075 11389 ts_tablet_manager.cc:1398] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:20.534384 11389 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:20.535902 11389 tablet_bootstrap.cc:654] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:20.538295 11389 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:20.538424 11389 ts_tablet_manager.cc:1367] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:17:20.539081 11389 raft_consensus.cc:348] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.539229 11389 raft_consensus.cc:374] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:20.539291 11389 raft_consensus.cc:729] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:20.539502 11389 consensus_queue.cc:260] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.539885 11389 ts_tablet_manager.cc:1398] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.001s	user 0.001s	sys 0.000s
I20240502 08:17:20.540195 11389 tablet_bootstrap.cc:492] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae: Bootstrap starting.
I20240502 08:17:20.541401 11389 tablet_bootstrap.cc:654] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:20.543028 11389 tablet_bootstrap.cc:492] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae: No bootstrap required, opened a new log
I20240502 08:17:20.543200 11389 ts_tablet_manager.cc:1367] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:17:20.543989 11389 raft_consensus.cc:348] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.544150 11389 raft_consensus.cc:374] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:20.544214 11389 raft_consensus.cc:729] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Initialized, Role: FOLLOWER
I20240502 08:17:20.544442 11389 consensus_queue.cc:260] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.544843 11389 ts_tablet_manager.cc:1398] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:17:20.621166 11330 raft_consensus.cc:482] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:20.621347 11330 raft_consensus.cc:504] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.622051 11330 leader_election.cc:288] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:20.628046 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a4038683ac48449ccad5eaeeef5694" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f577199118d5411b98b759d6c430daae" is_pre_election: true
I20240502 08:17:20.628291 10369 raft_consensus.cc:2454] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 0.
I20240502 08:17:20.628731 10433 leader_election.cc:302] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:20.629024 11330 raft_consensus.cc:2781] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:20.629133 11330 raft_consensus.cc:482] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:20.629226 11330 raft_consensus.cc:3037] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:20.631199 11330 raft_consensus.cc:504] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.631723 11330 leader_election.cc:288] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:20.632265 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a4038683ac48449ccad5eaeeef5694" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:20.632453 10369 raft_consensus.cc:3037] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:20.634353 10369 raft_consensus.cc:2454] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 1.
I20240502 08:17:20.634896 10433 leader_election.cc:302] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:20.635229 11330 raft_consensus.cc:2781] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:20.635381 11330 raft_consensus.cc:686] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Becoming Leader. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Running, Role: LEADER
I20240502 08:17:20.635643 11330 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.638743 10091 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67). New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:20.656445 11330 raft_consensus.cc:482] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:20.656656 11330 raft_consensus.cc:504] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.657145 11330 leader_election.cc:288] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:20.657842 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138b1526ab364e6bafe0abfa1edb4376" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f577199118d5411b98b759d6c430daae" is_pre_election: true
I20240502 08:17:20.658058 10369 raft_consensus.cc:2454] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 0.
I20240502 08:17:20.658445 10433 leader_election.cc:302] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:20.658766 11330 raft_consensus.cc:2781] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:20.658845 11330 raft_consensus.cc:482] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:20.658926 11330 raft_consensus.cc:3037] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:20.660804 11330 raft_consensus.cc:504] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.661289 11330 leader_election.cc:288] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:20.661810 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138b1526ab364e6bafe0abfa1edb4376" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:20.661985 10369 raft_consensus.cc:3037] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:20.664103 10369 raft_consensus.cc:2454] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 1.
I20240502 08:17:20.664547 10433 leader_election.cc:302] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:20.664851 11234 raft_consensus.cc:2781] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:20.665004 11234 raft_consensus.cc:686] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Becoming Leader. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Running, Role: LEADER
I20240502 08:17:20.665318 11234 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.669131 10091 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67). New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:20.677600 11311 raft_consensus.cc:482] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:20.677803 11311 raft_consensus.cc:504] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.678416 11311 leader_election.cc:288] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:20.679066 10236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d471c5fb3045495aa1ec0a774e6164d8" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02" is_pre_election: true
I20240502 08:17:20.679303 10236 raft_consensus.cc:2454] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 0.
I20240502 08:17:20.679848 10303 leader_election.cc:302] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:20.680142 11311 raft_consensus.cc:2781] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:17:20.680274 11311 raft_consensus.cc:482] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:20.680371 11311 raft_consensus.cc:3037] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:20.683187 11311 raft_consensus.cc:504] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.683717 11311 leader_election.cc:288] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:20.684368 10236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d471c5fb3045495aa1ec0a774e6164d8" candidate_uuid: "f577199118d5411b98b759d6c430daae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:20.684587 10236 raft_consensus.cc:3037] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:20.686834 10236 raft_consensus.cc:2454] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f577199118d5411b98b759d6c430daae in term 1.
I20240502 08:17:20.687245 10303 leader_election.cc:302] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 9279bdb860044c90ace75c2f03950e02, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:20.687559 11311 raft_consensus.cc:2781] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:20.687731 11311 raft_consensus.cc:686] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Leader. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:20.688042 11311 consensus_queue.cc:237] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } }
I20240502 08:17:20.690585 10091 catalog_manager.cc:5484] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae reported cstate change: term changed from 0 to 1, leader changed from <none> to f577199118d5411b98b759d6c430daae (127.6.151.66). New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } }
I20240502 08:17:20.783282 11234 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:20.783485 11234 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:20.784000 11234 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
W20240502 08:17:20.784922 10432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.68:37265: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240502 08:17:20.787931 10432 leader_election.cc:334] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Network error: Client connection negotiation failed: client connection to 127.6.151.68:37265: connect: Connection refused (error 111)
I20240502 08:17:20.788060 10432 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f; no voters: 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:20.788372 11234 raft_consensus.cc:2726] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 08:17:21.055608 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:21.055943 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:21.063544 11402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 08:17:21.137076 11234 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:21.195344 10237 raft_consensus.cc:1261] T d471c5fb3045495aa1ec0a774e6164d8 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:21.196146 11311 consensus_queue.cc:1035] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:21.199365 10369 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:17:21.200462 11234 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:17:21.283270 11427 mvcc.cc:205] Tried to move back new op lower bound from 7023156597545447424 to 7023156595366846464. Current Snapshot: MvccSnapshot[applied={T|T < 7023156597545447424 or (T in {7023156597545447424})}]
I20240502 08:17:21.310956 11437 mvcc.cc:205] Tried to move back new op lower bound from 7023156597526306816 to 7023156595459960832. Current Snapshot: MvccSnapshot[applied={T|T < 7023156597526306816}]

INSERT report
    rows total: 2000
    time total: 216.852 ms
  time per row: 0.108426 ms
I20240502 08:17:21.426481  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:37265
--local_ip_for_outbound_sockets=127.6.151.68
--tserver_master_addrs=127.6.151.126:36085
--webserver_port=43277
--webserver_interface=127.6.151.68
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:21.770323 11452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:21.770679 11452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:21.770850 11452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:21.783052 11452 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:21.783216 11452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:21.783392 11452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:17:21.797720 11452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:37265
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=43277
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:21.799041 11452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:21.800712 11452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:21.813130 11458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:21.813511 11457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:21.813887 11460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:21.814083 11452 server_base.cc:1034] running on GCE node
I20240502 08:17:21.814738 11452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:21.815760 11452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:21.816957 11452 hybrid_clock.cc:648] HybridClock initialized: now 1714637841816911 us; error 44 us; skew 500 ppm
I20240502 08:17:21.817265 11452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:21.819782 11452 webserver.cc:460] Webserver started at http://127.6.151.68:43277/ using document root <none> and password file <none>
I20240502 08:17:21.820513 11452 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:21.820642 11452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:21.826071 11452 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 08:17:21.829175 11466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:21.830674 11452 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:17:21.830847 11452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2fac43dcafe6465b941acc57014c43d3"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:21.831779 11452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:21.852993 11452 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:21.854331 11452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:21.854593 11452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:21.855636 11452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:21.857549 11473 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 08:17:21.863281 11452 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 08:17:21.863422 11452 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s	user 0.001s	sys 0.000s
I20240502 08:17:21.863575 11452 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 08:17:21.866490 11452 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 08:17:21.866627 11452 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s	user 0.000s	sys 0.000s
I20240502 08:17:21.867029 11473 tablet_bootstrap.cc:492] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap starting.
I20240502 08:17:21.893851 11473 log.cc:824] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:21.899461 11473 tablet_bootstrap.cc:492] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:21.900794 11473 tablet_bootstrap.cc:492] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap complete.
I20240502 08:17:21.902549 11473 ts_tablet_manager.cc:1367] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.021s
I20240502 08:17:21.903832 11452 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:37265
I20240502 08:17:21.903879 11580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:37265 every 8 connection(s)
I20240502 08:17:21.905627 11452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:17:21.911104  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 11452
I20240502 08:17:21.911726  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:44733
--local_ip_for_outbound_sockets=127.6.151.69
--tserver_master_addrs=127.6.151.126:36085
--webserver_port=43663
--webserver_interface=127.6.151.69
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:21.912338 11473 raft_consensus.cc:348] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:21.913602 11473 raft_consensus.cc:729] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Initialized, Role: FOLLOWER
I20240502 08:17:21.914327 11473 consensus_queue.cc:260] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:21.916085 11473 ts_tablet_manager.cc:1398] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.001s
I20240502 08:17:21.920982 11581 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:21.921298 11581 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:21.922174 11581 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:21.924676 10096 ts_manager.cc:194] Re-registered known tserver with Master: 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
I20240502 08:17:21.925825 10096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:52713
I20240502 08:17:21.928823 11581 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
W20240502 08:17:22.157680 11582 tablet.cc:2369] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:17:22.265480 11586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:22.265810 11586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:22.265897 11586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:22.273308 11586 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:22.273447 11586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:22.273559 11586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:17:22.282835 11586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:44733
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=43663
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:22.284130 11586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:22.285784 11586 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:22.298421 11592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:22.298537 11591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:22.298846 11594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:22.299095 11586 server_base.cc:1034] running on GCE node
I20240502 08:17:22.299750 11586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:22.300720 11586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:22.301954 11586 hybrid_clock.cc:648] HybridClock initialized: now 1714637842301883 us; error 44 us; skew 500 ppm
I20240502 08:17:22.302313 11586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:22.311072 11586 webserver.cc:460] Webserver started at http://127.6.151.69:43663/ using document root <none> and password file <none>
I20240502 08:17:22.311921 11586 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:22.312053 11586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:22.320287 11586 fs_manager.cc:709] Time spent opening directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 08:17:22.323652 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:22.325134 11586 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:17:22.325313 11586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "54f96b9fb77540e898821b386cf6d0cb"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:22.326224 11586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:22.354280 11330 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:22.354519 11330 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:22.355120 11330 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
I20240502 08:17:22.363194 11535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f8034ce87a4c85a72c53ba07f178d1" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fac43dcafe6465b941acc57014c43d3" is_pre_election: true
I20240502 08:17:22.363771 11535 raft_consensus.cc:2396] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 750dbb32b0114697b81aeb7fca556e7f for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 08:17:22.364909 10432 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f; no voters: 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:22.365258 11330 raft_consensus.cc:2726] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:22.370604 11586 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:22.371513 11586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:22.371814 11586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:22.372781 11586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:22.374306 11586 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:22.374387 11586 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:22.374521 11586 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:22.374596 11586 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:22.406963 11586 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:44733
I20240502 08:17:22.406975 11714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:44733 every 8 connection(s)
I20240502 08:17:22.408488 11586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:17:22.410012  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 11586
I20240502 08:17:22.410624  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.70:33091
--local_ip_for_outbound_sockets=127.6.151.70
--tserver_master_addrs=127.6.151.126:36085
--webserver_port=38285
--webserver_interface=127.6.151.70
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:22.423626 11715 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:22.423969 11715 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:22.424876 11715 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:22.426386 10093 ts_manager.cc:194] Re-registered known tserver with Master: 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733)
I20240502 08:17:22.427302 10093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:59611
W20240502 08:17:22.758291 11718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:22.758608 11718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:22.758783 11718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:22.766476 11718 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:22.766626 11718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:22.766819 11718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.70
I20240502 08:17:22.776006 11718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.70:33091
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.151.70
--webserver_port=38285
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.70
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:22.777422 11718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:22.779232 11718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:22.791059 11723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:22.791240 11724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:22.791438 11726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:22.792258 11718 server_base.cc:1034] running on GCE node
I20240502 08:17:22.792994 11718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:22.794183 11718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:22.795398 11718 hybrid_clock.cc:648] HybridClock initialized: now 1714637842795318 us; error 78 us; skew 500 ppm
I20240502 08:17:22.795730 11718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:22.797955 11718 webserver.cc:460] Webserver started at http://127.6.151.70:38285/ using document root <none> and password file <none>
I20240502 08:17:22.798715 11718 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:22.798844 11718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:22.804070 11718 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:17:22.806936 11732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:22.808475 11718 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:17:22.808668 11718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a57c9c479b194a69b5eb8f9361819a83"
format_stamp: "Formatted at 2024-05-02 08:17:10 on dist-test-slave-x3qn"
I20240502 08:17:22.809520 11718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:22.871220 11718 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:22.872084 11718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:22.872375 11718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:22.873332 11718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:22.874960 11718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:22.875042 11718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:22.875186 11718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:22.875242 11718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:22.908458 11718 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.70:33091
I20240502 08:17:22.908464 11844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.70:33091 every 8 connection(s)
I20240502 08:17:22.910348 11718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:17:22.919430  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 11718
I20240502 08:17:22.919989  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.71:43045
--local_ip_for_outbound_sockets=127.6.151.71
--tserver_master_addrs=127.6.151.126:36085
--webserver_port=36385
--webserver_interface=127.6.151.71
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:22.923972 11845 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:22.924310 11845 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:22.925179 11845 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:22.926774 10093 ts_manager.cc:194] Re-registered known tserver with Master: a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70:33091)
I20240502 08:17:22.927904 10093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.70:60615
I20240502 08:17:23.077059 11849 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:17:23.077317 11849 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:23.078833 11849 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:23.086459 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f8034ce87a4c85a72c53ba07f178d1" candidate_uuid: "2fac43dcafe6465b941acc57014c43d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f" is_pre_election: true
I20240502 08:17:23.086732 10499 raft_consensus.cc:2454] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fac43dcafe6465b941acc57014c43d3 in term 1.
I20240502 08:17:23.087208 11469 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2fac43dcafe6465b941acc57014c43d3, 750dbb32b0114697b81aeb7fca556e7f; no voters: 
I20240502 08:17:23.087568 11849 raft_consensus.cc:2781] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 08:17:23.087729 11849 raft_consensus.cc:482] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:17:23.087822 11849 raft_consensus.cc:3037] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:23.091428 11849 raft_consensus.cc:504] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:23.092007 11849 leader_election.cc:288] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 2 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:23.092515 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f8034ce87a4c85a72c53ba07f178d1" candidate_uuid: "2fac43dcafe6465b941acc57014c43d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:23.092773 10499 raft_consensus.cc:3037] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:23.094625 10499 raft_consensus.cc:2454] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fac43dcafe6465b941acc57014c43d3 in term 2.
I20240502 08:17:23.095090 11469 leader_election.cc:302] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2fac43dcafe6465b941acc57014c43d3, 750dbb32b0114697b81aeb7fca556e7f; no voters: 
I20240502 08:17:23.095435 11849 raft_consensus.cc:2781] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:23.095885 11849 raft_consensus.cc:686] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [term 2 LEADER]: Becoming Leader. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Running, Role: LEADER
I20240502 08:17:23.096311 11849 consensus_queue.cc:237] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } }
I20240502 08:17:23.100515 10093 catalog_manager.cc:5484] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2fac43dcafe6465b941acc57014c43d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:23.272398 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:23.272723 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:23.272810 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:23.281309 11848 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:23.281445 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:23.281565 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.71
I20240502 08:17:23.290863 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.71:43045
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.151.71
--webserver_port=36385
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.71
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:23.292199 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:23.293996 11848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:23.306273 11862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:23.306474 11865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:23.306598 11863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:23.306978 11848 server_base.cc:1034] running on GCE node
I20240502 08:17:23.307569 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:23.308532 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:23.309738 11848 hybrid_clock.cc:648] HybridClock initialized: now 1714637843309693 us; error 47 us; skew 500 ppm
I20240502 08:17:23.310072 11848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:23.312484 11848 webserver.cc:460] Webserver started at http://127.6.151.71:36385/ using document root <none> and password file <none>
I20240502 08:17:23.313300 11848 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:23.313473 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:23.318969 11848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:17:23.321997 11871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:23.323403 11848 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:17:23.323563 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "84379f2079ff403eacad196d57abd7ad"
format_stamp: "Formatted at 2024-05-02 08:17:11 on dist-test-slave-x3qn"
I20240502 08:17:23.324362 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:23.341353 11848 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:23.342196 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:23.342486 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:23.343499 11848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:23.345055 11848 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:23.345129 11848 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:23.345247 11848 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:23.345297 11848 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:23.376638 11848 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.71:43045
I20240502 08:17:23.376698 11983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.71:43045 every 8 connection(s)
I20240502 08:17:23.378067 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:17:23.386503  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 11848
I20240502 08:17:23.387009  6749 test_util.cc:268] Using random seed: 999477831
I20240502 08:17:23.391379 11984 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:23.391686 11984 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:23.392544 11984 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:23.393909 10093 ts_manager.cc:194] Re-registered known tserver with Master: 84379f2079ff403eacad196d57abd7ad (127.6.151.71:43045)
I20240502 08:17:23.394814 10093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.71:58517
I20240502 08:17:23.402786  6749 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 08:17:23.403260  6749 test_util.cc:268] Using random seed: 999494106
I20240502 08:17:23.424804  6749 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 08:17:23.425592  6749 test_util.cc:268] Using random seed: 999516411
I20240502 08:17:23.430040 11715 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
I20240502 08:17:23.459131  6749 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 08:17:23.459508  6749 test_util.cc:268] Using random seed: 999550349
I20240502 08:17:23.505367  6749 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 08:17:23.508944  6749 test_util.cc:268] Using random seed: 999599775
I20240502 08:17:23.535338 10499 raft_consensus.cc:1261] T f5f8034ce87a4c85a72c53ba07f178d1 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Refusing update from remote peer 2fac43dcafe6465b941acc57014c43d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 08:17:23.536227 11849 consensus_queue.cc:1035] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:17:23.564477 12021 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.6.151.126:36085, user_credentials={real_user=slave}} blocked reactor thread for 49204.9us
I20240502 08:17:23.587564  6749 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240502 08:17:23.670327 12040 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.6.151.126:36085, user_credentials={real_user=slave}} blocked reactor thread for 43674.3us
I20240502 08:17:23.930660 11845 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
I20240502 08:17:24.397408 11984 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
I20240502 08:17:24.888120 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:24.981966 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:25.015725 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:25.033391 11919 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:25.036506 11780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:25.064584 10348 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:25.079056 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:25.876386 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:26.000903 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:26.032711 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:26.048194 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:26.067889 11919 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:26.111264 11780 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:26.228446 10348 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:17:26.496949 10496 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:26.506531 10368 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 08:17:26.508919 12080 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240502 08:17:26.514797 10369 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:26.515815 12057 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:26.524073 10093 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 65, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
W20240502 08:17:26.529757 10435 consensus_peers.cc:485] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f -> Peer a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70:33091): Couldn't send request to peer a57c9c479b194a69b5eb8f9361819a83. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0a4038683ac48449ccad5eaeeef5694. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:26.640014 10368 consensus_queue.cc:237] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:26.646922 10235 raft_consensus.cc:1261] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 08:17:26.648397 12149 consensus_queue.cc:1035] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 08:17:26.665246 12028 raft_consensus.cc:2932] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:26.671645 10093 catalog_manager.cc:5484] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 66, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:26.672556 10237 raft_consensus.cc:2932] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
W20240502 08:17:26.703979 10303 consensus_peers.cc:485] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae -> Peer 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733): Couldn't send request to peer 54f96b9fb77540e898821b386cf6d0cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb8df38d1c3340fda6024301f0798487. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:26.800263 10237 consensus_queue.cc:237] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 1.55, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:26.806174 10499 raft_consensus.cc:1261] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 55. Preceding OpId from leader: term: 1 index: 56. (index mismatch)
I20240502 08:17:26.828531 12158 consensus_queue.cc:1035] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 56, Last known committed idx: 55, Time since last communication: 0.000s
W20240502 08:17:26.831804 10174 consensus_peers.cc:485] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 -> Peer 84379f2079ff403eacad196d57abd7ad (127.6.151.71:43045): Couldn't send request to peer 84379f2079ff403eacad196d57abd7ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: befc5d75417d47d8b7790017c49fcb71. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:26.836335 11225 raft_consensus.cc:2932] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:26.856241 10093 catalog_manager.cc:5484] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index -1 to 56, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:26.871752 10499 raft_consensus.cc:2932] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.56: config changed from index -1 to 56, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:26.952585 10238 consensus_queue.cc:237] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:26.963793 10368 raft_consensus.cc:1261] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240502 08:17:26.965061 12158 consensus_queue.cc:1035] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240502 08:17:26.970934 11225 raft_consensus.cc:2932] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } } }
I20240502 08:17:26.974722 10368 raft_consensus.cc:2932] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } } }
W20240502 08:17:26.979564 10171 consensus_peers.cc:485] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 -> Peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Couldn't send request to peer 2fac43dcafe6465b941acc57014c43d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d426a029b71e4dd4a8852a5cfe708ae0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:26.980122 10096 catalog_manager.cc:5484] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 63, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } } }
I20240502 08:17:27.079560 12230 ts_tablet_manager.cc:902] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: Initiating tablet copy from peer 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:27.085264 10496 consensus_queue.cc:237] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:27.092159 10235 raft_consensus.cc:1261] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240502 08:17:27.092831 11330 consensus_queue.cc:1035] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
W20240502 08:17:27.093113 10435 consensus_peers.cc:485] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f -> Peer a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70:33091): Couldn't send request to peer a57c9c479b194a69b5eb8f9361819a83. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82835c215cce4ce49680d56a5bbf0208. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:27.097219 12080 raft_consensus.cc:2932] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:27.098418 10237 raft_consensus.cc:2932] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:27.103144 10093 catalog_manager.cc:5484] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index -1 to 83, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:27.105616 12233 ts_tablet_manager.cc:902] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:27.118593 12230 tablet_copy_client.cc:308] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:38465
I20240502 08:17:27.151425 12233 tablet_copy_client.cc:308] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:27.158080 10519 tablet_copy_service.cc:140] P 750dbb32b0114697b81aeb7fca556e7f: Received BeginTabletCopySession request for tablet c0a4038683ac48449ccad5eaeeef5694 from peer a57c9c479b194a69b5eb8f9361819a83 ({username='slave'} at 127.6.151.70:46585)
I20240502 08:17:27.158380 10519 tablet_copy_service.cc:161] P 750dbb32b0114697b81aeb7fca556e7f: Beginning new tablet copy session on tablet c0a4038683ac48449ccad5eaeeef5694 from peer a57c9c479b194a69b5eb8f9361819a83 at {username='slave'} at 127.6.151.70:46585: session id = a57c9c479b194a69b5eb8f9361819a83-c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:27.162081 10519 tablet_copy_source_session.cc:205] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:27.175797 12230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a4038683ac48449ccad5eaeeef5694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.194115 12230 tablet_copy_client.cc:783] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:27.194626 12230 tablet_copy_client.cc:647] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:27.201996 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet fb8df38d1c3340fda6024301f0798487 from peer 54f96b9fb77540e898821b386cf6d0cb ({username='slave'} at 127.6.151.69:59351)
I20240502 08:17:27.202265 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet fb8df38d1c3340fda6024301f0798487 from peer 54f96b9fb77540e898821b386cf6d0cb at {username='slave'} at 127.6.151.69:59351: session id = 54f96b9fb77540e898821b386cf6d0cb-fb8df38d1c3340fda6024301f0798487
I20240502 08:17:27.203423 12230 tablet_copy_client.cc:523] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:27.209405 12230 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap starting.
I20240502 08:17:27.210768 10389 tablet_copy_source_session.cc:205] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:27.222301 12233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8df38d1c3340fda6024301f0798487. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.240289 12233 tablet_copy_client.cc:783] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:27.240743 12233 tablet_copy_client.cc:647] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:27.256772 12230 log.cc:824] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:27.268458 12233 tablet_copy_client.cc:523] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:27.342583 12233 tablet_bootstrap.cc:492] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap starting.
I20240502 08:17:27.347030 10369 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
W20240502 08:17:27.350150 10303 consensus_peers.cc:485] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae -> Peer 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733): Couldn't send request to peer 54f96b9fb77540e898821b386cf6d0cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b27b7cc14814dc48c1fe37dbbea9619. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:27.365491 10499 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240502 08:17:27.372123 12149 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240502 08:17:27.402398 12081 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:27.428323 10096 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 83, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:27.429600 10497 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index -1 to 83, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:27.476162 12252 ts_tablet_manager.cc:902] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: Initiating tablet copy from peer 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:27.515236 12252 tablet_copy_client.cc:308] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43271
I20240502 08:17:27.532285 12233 log.cc:824] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:27.526063 12251 ts_tablet_manager.cc:902] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: Initiating tablet copy from peer 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:27.535079 10258 tablet_copy_service.cc:140] P 9279bdb860044c90ace75c2f03950e02: Received BeginTabletCopySession request for tablet befc5d75417d47d8b7790017c49fcb71 from peer 84379f2079ff403eacad196d57abd7ad ({username='slave'} at 127.6.151.71:60835)
I20240502 08:17:27.535405 10258 tablet_copy_service.cc:161] P 9279bdb860044c90ace75c2f03950e02: Beginning new tablet copy session on tablet befc5d75417d47d8b7790017c49fcb71 from peer 84379f2079ff403eacad196d57abd7ad at {username='slave'} at 127.6.151.71:60835: session id = 84379f2079ff403eacad196d57abd7ad-befc5d75417d47d8b7790017c49fcb71
I20240502 08:17:27.540951 10258 tablet_copy_source_session.cc:205] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:27.549162 12252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befc5d75417d47d8b7790017c49fcb71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.560223 10368 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:27.562765 10235 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 1 index: 66. (index mismatch)
I20240502 08:17:27.564389 12004 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240502 08:17:27.567366 12004 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:27.567956 10235 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.66: config changed from index -1 to 66, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:27.577960 10096 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 66, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:27.581421 12252 tablet_copy_client.cc:783] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:27.581845 12252 tablet_copy_client.cc:647] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:27.594108 12252 tablet_copy_client.cc:523] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:27.599947 12252 tablet_bootstrap.cc:492] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: Bootstrap starting.
I20240502 08:17:27.613219 12251 tablet_copy_client.cc:308] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43271
I20240502 08:17:27.640739 10258 tablet_copy_service.cc:140] P 9279bdb860044c90ace75c2f03950e02: Received BeginTabletCopySession request for tablet d426a029b71e4dd4a8852a5cfe708ae0 from peer 2fac43dcafe6465b941acc57014c43d3 ({username='slave'} at 127.6.151.68:60407)
I20240502 08:17:27.640961 10258 tablet_copy_service.cc:161] P 9279bdb860044c90ace75c2f03950e02: Beginning new tablet copy session on tablet d426a029b71e4dd4a8852a5cfe708ae0 from peer 2fac43dcafe6465b941acc57014c43d3 at {username='slave'} at 127.6.151.68:60407: session id = 2fac43dcafe6465b941acc57014c43d3-d426a029b71e4dd4a8852a5cfe708ae0
I20240502 08:17:27.643821 10258 tablet_copy_source_session.cc:205] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:27.647560 12251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d426a029b71e4dd4a8852a5cfe708ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.656780 12251 tablet_copy_client.cc:783] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:27.657218 12251 tablet_copy_client.cc:647] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:27.660286 12251 tablet_copy_client.cc:523] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:27.665984 12251 tablet_bootstrap.cc:492] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap starting.
I20240502 08:17:27.671041 12230 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=74 ignored=0} inserts{seen=705 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:27.674171 12230 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap complete.
I20240502 08:17:27.675805 12252 log.cc:824] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:27.676554 12230 ts_tablet_manager.cc:1367] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: Time spent bootstrapping tablet: real 0.467s	user 0.161s	sys 0.066s
W20240502 08:17:27.690048 10433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43271, user_credentials={real_user=slave}} blocked reactor thread for 61257us
I20240502 08:17:27.700508 12230 raft_consensus.cc:348] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
W20240502 08:17:27.704972 10305 consensus_peers.cc:485] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae -> Peer 84379f2079ff403eacad196d57abd7ad (127.6.151.71:43045): Couldn't send request to peer 84379f2079ff403eacad196d57abd7ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd1419d08a94610b725cc1ed2b04d00. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:27.717998 12230 raft_consensus.cc:729] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a57c9c479b194a69b5eb8f9361819a83, State: Initialized, Role: LEARNER
I20240502 08:17:27.719082 12230 consensus_queue.cc:260] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:27.730441 12274 ts_tablet_manager.cc:902] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: Initiating tablet copy from peer 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:27.737341 12230 ts_tablet_manager.cc:1398] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83: Time spent starting tablet: real 0.061s	user 0.007s	sys 0.004s
I20240502 08:17:27.748726 10519 tablet_copy_service.cc:342] P 750dbb32b0114697b81aeb7fca556e7f: Request end of tablet copy session a57c9c479b194a69b5eb8f9361819a83-c0a4038683ac48449ccad5eaeeef5694 received from {username='slave'} at 127.6.151.70:46585
I20240502 08:17:27.748945 10519 tablet_copy_service.cc:433] P 750dbb32b0114697b81aeb7fca556e7f: ending tablet copy session a57c9c479b194a69b5eb8f9361819a83-c0a4038683ac48449ccad5eaeeef5694 on tablet c0a4038683ac48449ccad5eaeeef5694 with peer a57c9c479b194a69b5eb8f9361819a83
I20240502 08:17:27.846988 12251 tablet_bootstrap.cc:492] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=73 ignored=0} inserts{seen=745 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:27.847615 12251 tablet_bootstrap.cc:492] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap complete.
I20240502 08:17:27.847962 12251 ts_tablet_manager.cc:1367] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: Time spent bootstrapping tablet: real 0.182s	user 0.132s	sys 0.030s
I20240502 08:17:27.850267 12251 raft_consensus.cc:348] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:27.884913 12251 raft_consensus.cc:729] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Initialized, Role: LEARNER
I20240502 08:17:27.885339 12251 consensus_queue.cc:260] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 73, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:27.892976 12251 ts_tablet_manager.cc:1398] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3: Time spent starting tablet: real 0.045s	user 0.025s	sys 0.016s
I20240502 08:17:27.894584 10258 tablet_copy_service.cc:342] P 9279bdb860044c90ace75c2f03950e02: Request end of tablet copy session 2fac43dcafe6465b941acc57014c43d3-d426a029b71e4dd4a8852a5cfe708ae0 received from {username='slave'} at 127.6.151.68:60407
I20240502 08:17:27.894809 10258 tablet_copy_service.cc:433] P 9279bdb860044c90ace75c2f03950e02: ending tablet copy session 2fac43dcafe6465b941acc57014c43d3-d426a029b71e4dd4a8852a5cfe708ae0 on tablet d426a029b71e4dd4a8852a5cfe708ae0 with peer 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:27.906162 12281 ts_tablet_manager.cc:902] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:27.927037 12274 tablet_copy_client.cc:308] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:38465
I20240502 08:17:27.927775 10519 tablet_copy_service.cc:140] P 750dbb32b0114697b81aeb7fca556e7f: Received BeginTabletCopySession request for tablet 82835c215cce4ce49680d56a5bbf0208 from peer a57c9c479b194a69b5eb8f9361819a83 ({username='slave'} at 127.6.151.70:46585)
I20240502 08:17:27.927958 10519 tablet_copy_service.cc:161] P 750dbb32b0114697b81aeb7fca556e7f: Beginning new tablet copy session on tablet 82835c215cce4ce49680d56a5bbf0208 from peer a57c9c479b194a69b5eb8f9361819a83 at {username='slave'} at 127.6.151.70:46585: session id = a57c9c479b194a69b5eb8f9361819a83-82835c215cce4ce49680d56a5bbf0208
I20240502 08:17:27.930949 10519 tablet_copy_source_session.cc:205] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:27.932633 12274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82835c215cce4ce49680d56a5bbf0208. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:27.939358 12274 tablet_copy_client.cc:783] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:27.939724 12274 tablet_copy_client.cc:647] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:27.943709 12274 tablet_copy_client.cc:523] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:27.950078 12274 tablet_bootstrap.cc:492] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap starting.
I20240502 08:17:27.970852 10238 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
W20240502 08:17:27.974543 10171 consensus_peers.cc:485] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 -> Peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Couldn't send request to peer 2fac43dcafe6465b941acc57014c43d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00d1cef3b67e4d58aa76c8fd23f2c099. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:27.975533 10499 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 08:17:27.976419 12224 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 92, Time since last communication: 0.000s
I20240502 08:17:27.978729 12252 tablet_bootstrap.cc:492] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=755 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:27.979341 12252 tablet_bootstrap.cc:492] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: Bootstrap complete.
I20240502 08:17:27.979038 12224 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index -1 to 93, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } } }
I20240502 08:17:27.979849 12252 ts_tablet_manager.cc:1367] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: Time spent bootstrapping tablet: real 0.380s	user 0.144s	sys 0.010s
I20240502 08:17:27.983474 10497 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index -1 to 93, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } } }
I20240502 08:17:27.987474 12252 raft_consensus.cc:348] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:27.988236 12252 raft_consensus.cc:729] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84379f2079ff403eacad196d57abd7ad, State: Initialized, Role: LEARNER
I20240502 08:17:27.988865 12252 consensus_queue.cc:260] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 56 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:27.991890 10096 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index -1 to 93, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:28.045164 11800 raft_consensus.cc:1203] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.64->[1.65-1.84]   Dedup: 1.74->[1.75-1.84]
I20240502 08:17:28.053048 10368 consensus_queue.cc:237] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.91, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:28.054963 12302 ts_tablet_manager.cc:902] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:28.067890 12281 tablet_copy_client.cc:308] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:28.072975 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet 2b27b7cc14814dc48c1fe37dbbea9619 from peer 54f96b9fb77540e898821b386cf6d0cb ({username='slave'} at 127.6.151.69:59351)
I20240502 08:17:28.073367 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet 2b27b7cc14814dc48c1fe37dbbea9619 from peer 54f96b9fb77540e898821b386cf6d0cb at {username='slave'} at 127.6.151.69:59351: session id = 54f96b9fb77540e898821b386cf6d0cb-2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:28.073640 10238 raft_consensus.cc:1261] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 92. (index mismatch)
I20240502 08:17:28.077731 10389 tablet_copy_source_session.cc:205] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:28.082864 12281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b27b7cc14814dc48c1fe37dbbea9619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:28.115891 12187 consensus_queue.cc:1035] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 92, Last known committed idx: 91, Time since last communication: 0.000s
I20240502 08:17:28.133582 12057 raft_consensus.cc:1050] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: attempting to promote NON_VOTER a57c9c479b194a69b5eb8f9361819a83 to VOTER
I20240502 08:17:28.134517 12057 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:28.137220 11800 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 08:17:28.137987 12080 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 08:17:28.142335 11330 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index 65 to 86, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.147297 11800 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 65 to 86, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.157285 10096 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 reported cstate change: config changed from index 65 to 86, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.165470 10368 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 08:17:28.169203 12080 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 08:17:28.187273 12028 raft_consensus.cc:2932] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:28.207077 10497 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.89, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:28.209244 11800 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 90. (index mismatch)
I20240502 08:17:28.210494 10368 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 65 to 86, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.216241 12080 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 90, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 08:17:28.219365 12079 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.90: config changed from index 86 to 90, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.223556 10083 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0a4038683ac48449ccad5eaeeef5694 with cas_config_opid_index 86: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:28.218391 10096 catalog_manager.cc:5484] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 92, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:28.224802 11800 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.90: config changed from index 86 to 90, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.232586 10093 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 reported cstate change: config changed from index 86 to 90, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 90 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.236907 12274 tablet_bootstrap.cc:492] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=765 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:28.237509 12274 tablet_bootstrap.cc:492] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap complete.
I20240502 08:17:28.237716 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet c0a4038683ac48449ccad5eaeeef5694 with delete_type TABLET_DATA_TOMBSTONED (TS f577199118d5411b98b759d6c430daae not found in new config with opid_index 90) from {username='slave'} at 127.0.0.1:52194
I20240502 08:17:28.237850 12274 ts_tablet_manager.cc:1367] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: Time spent bootstrapping tablet: real 0.288s	user 0.157s	sys 0.008s
I20240502 08:17:28.238950 12274 raft_consensus.cc:348] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:28.239454 12274 raft_consensus.cc:729] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a57c9c479b194a69b5eb8f9361819a83, State: Initialized, Role: LEARNER
I20240502 08:17:28.239732 12274 consensus_queue.cc:260] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:28.240689 12274 ts_tablet_manager.cc:1398] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:17:28.241503 10519 tablet_copy_service.cc:342] P 750dbb32b0114697b81aeb7fca556e7f: Request end of tablet copy session a57c9c479b194a69b5eb8f9361819a83-82835c215cce4ce49680d56a5bbf0208 received from {username='slave'} at 127.6.151.70:46585
I20240502 08:17:28.241634 10519 tablet_copy_service.cc:433] P 750dbb32b0114697b81aeb7fca556e7f: ending tablet copy session a57c9c479b194a69b5eb8f9361819a83-82835c215cce4ce49680d56a5bbf0208 on tablet 82835c215cce4ce49680d56a5bbf0208 with peer a57c9c479b194a69b5eb8f9361819a83
I20240502 08:17:28.243359 12313 tablet_replica.cc:330] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: stopping tablet replica
I20240502 08:17:28.246337 10235 raft_consensus.cc:2932] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.92: config changed from index -1 to 92, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:28.247292 12313 raft_consensus.cc:2229] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:28.249200 12313 raft_consensus.cc:2258] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:28.253216 12313 ts_tablet_manager.cc:1875] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:28.289685 12313 ts_tablet_manager.cc:1888] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.86
I20240502 08:17:28.303474 12313 log.cc:1196] T c0a4038683ac48449ccad5eaeeef5694 P f577199118d5411b98b759d6c430daae: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:28.305370 10082 catalog_manager.cc:4830] TS f577199118d5411b98b759d6c430daae (127.6.151.66:41685): tablet c0a4038683ac48449ccad5eaeeef5694 (table rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]) successfully deleted
W20240502 08:17:28.313411 10305 consensus_peers.cc:485] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae -> Peer a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70:33091): Couldn't send request to peer a57c9c479b194a69b5eb8f9361819a83. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21631366c38a4a608cb8191903fc535d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:28.343137 12281 tablet_copy_client.cc:783] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:28.343585 12281 tablet_copy_client.cc:647] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:28.349956 11535 raft_consensus.cc:1203] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.62->[1.63-1.79]   Dedup: 1.74->[1.75-1.79]
I20240502 08:17:28.378994 12252 ts_tablet_manager.cc:1398] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad: Time spent starting tablet: real 0.399s	user 0.156s	sys 0.199s
I20240502 08:17:28.382973 10258 tablet_copy_service.cc:342] P 9279bdb860044c90ace75c2f03950e02: Request end of tablet copy session 84379f2079ff403eacad196d57abd7ad-befc5d75417d47d8b7790017c49fcb71 received from {username='slave'} at 127.6.151.71:60835
I20240502 08:17:28.383123 10258 tablet_copy_service.cc:433] P 9279bdb860044c90ace75c2f03950e02: ending tablet copy session 84379f2079ff403eacad196d57abd7ad-befc5d75417d47d8b7790017c49fcb71 on tablet befc5d75417d47d8b7790017c49fcb71 with peer 84379f2079ff403eacad196d57abd7ad
I20240502 08:17:28.392869 12302 tablet_copy_client.cc:308] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:28.414767 12281 tablet_copy_client.cc:523] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:28.419993 11939 raft_consensus.cc:1203] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.55->[1.56-1.78]   Dedup: 1.70->[1.71-1.78]
I20240502 08:17:28.433307 12281 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap starting.
I20240502 08:17:28.455019 12329 ts_tablet_manager.cc:902] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: Initiating tablet copy from peer 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:28.485282 12329 tablet_copy_client.cc:308] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43271
I20240502 08:17:28.486127 10258 tablet_copy_service.cc:140] P 9279bdb860044c90ace75c2f03950e02: Received BeginTabletCopySession request for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 from peer 2fac43dcafe6465b941acc57014c43d3 ({username='slave'} at 127.6.151.68:60407)
I20240502 08:17:28.486289 10258 tablet_copy_service.cc:161] P 9279bdb860044c90ace75c2f03950e02: Beginning new tablet copy session on tablet 00d1cef3b67e4d58aa76c8fd23f2c099 from peer 2fac43dcafe6465b941acc57014c43d3 at {username='slave'} at 127.6.151.68:60407: session id = 2fac43dcafe6465b941acc57014c43d3-00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:28.490190 10258 tablet_copy_source_session.cc:205] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:28.493256 12329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d1cef3b67e4d58aa76c8fd23f2c099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:28.504058 12224 raft_consensus.cc:1050] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02: attempting to promote NON_VOTER 84379f2079ff403eacad196d57abd7ad to VOTER
I20240502 08:17:28.505362 12224 consensus_queue.cc:237] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:28.509460 10497 raft_consensus.cc:1261] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240502 08:17:28.509692 12329 tablet_copy_client.cc:783] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:28.509968 12329 tablet_copy_client.cc:647] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:28.510428 11939 raft_consensus.cc:1261] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240502 08:17:28.511096 12274 ts_tablet_manager.cc:902] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:28.511171 12148 consensus_queue.cc:1035] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 08:17:28.511853 12148 consensus_queue.cc:1035] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 78, Time since last communication: 0.000s
I20240502 08:17:28.516355 12329 tablet_copy_client.cc:523] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:28.517601 10497 raft_consensus.cc:2932] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 56 to 80, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:28.519919 12329 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap starting.
I20240502 08:17:28.523836 10093 catalog_manager.cc:5484] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 56 to 80, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:28.524119 12300 raft_consensus.cc:2932] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 56 to 80, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:28.524698 11939 raft_consensus.cc:2932] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 56 to 80, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:28.547755 12274 tablet_copy_client.cc:308] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:28.566095 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet 6fd1419d08a94610b725cc1ed2b04d00 from peer 84379f2079ff403eacad196d57abd7ad ({username='slave'} at 127.6.151.71:49841)
I20240502 08:17:28.566362 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet 6fd1419d08a94610b725cc1ed2b04d00 from peer 84379f2079ff403eacad196d57abd7ad at {username='slave'} at 127.6.151.71:49841: session id = 84379f2079ff403eacad196d57abd7ad-6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:28.567351 10368 consensus_queue.cc:237] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 90, Committed index: 90, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:28.570778 10389 tablet_copy_source_session.cc:205] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:28.579515 10238 consensus_queue.cc:237] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:28.581241 11939 raft_consensus.cc:1261] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 08:17:28.581938 12148 consensus_queue.cc:1035] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 08:17:28.584843 11939 raft_consensus.cc:2932] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 80 to 82, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:28.590077 10093 catalog_manager.cc:5484] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad reported cstate change: config changed from index 80 to 82, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:28.590811 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet 21631366c38a4a608cb8191903fc535d from peer a57c9c479b194a69b5eb8f9361819a83 ({username='slave'} at 127.6.151.70:50673)
I20240502 08:17:28.590988 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet 21631366c38a4a608cb8191903fc535d from peer a57c9c479b194a69b5eb8f9361819a83 at {username='slave'} at 127.6.151.70:50673: session id = a57c9c479b194a69b5eb8f9361819a83-21631366c38a4a608cb8191903fc535d
I20240502 08:17:28.594082 10389 tablet_copy_source_session.cc:205] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:28.593920 12300 raft_consensus.cc:2932] T befc5d75417d47d8b7790017c49fcb71 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 80 to 82, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:28.596346 12274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21631366c38a4a608cb8191903fc535d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:28.600715 10479 tablet_service.cc:1514] Processing DeleteTablet for tablet befc5d75417d47d8b7790017c49fcb71 with delete_type TABLET_DATA_TOMBSTONED (TS 750dbb32b0114697b81aeb7fca556e7f not found in new config with opid_index 82) from {username='slave'} at 127.0.0.1:46754
I20240502 08:17:28.602452 12274 tablet_copy_client.cc:783] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:28.602787 12274 tablet_copy_client.cc:647] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:28.604943 12274 tablet_copy_client.cc:523] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:28.608724 12274 tablet_bootstrap.cc:492] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: Bootstrap starting.
I20240502 08:17:28.610714 10238 raft_consensus.cc:1261] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 08:17:28.612023 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd1419d08a94610b725cc1ed2b04d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:28.619110 12302 tablet_copy_client.cc:783] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:28.619439 12302 tablet_copy_client.cc:647] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:28.626188 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet befc5d75417d47d8b7790017c49fcb71 with cas_config_opid_index 80: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:28.632519 12348 tablet_replica.cc:330] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: stopping tablet replica
I20240502 08:17:28.633141 12348 raft_consensus.cc:2229] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:28.633479 12348 raft_consensus.cc:2258] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:28.637301 12081 consensus_queue.cc:1035] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 90, Time since last communication: 0.000s
W20240502 08:17:28.642199 10303 consensus_peers.cc:485] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae -> Peer 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733): Couldn't send request to peer 54f96b9fb77540e898821b386cf6d0cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75f37a6f3a254fec8b7805185d9ae80f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:28.643151 12302 tablet_copy_client.cc:523] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:28.643162 12348 ts_tablet_manager.cc:1875] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:28.646778 12302 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: Bootstrap starting.
I20240502 08:17:28.661489 12348 ts_tablet_manager.cc:1888] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.81
I20240502 08:17:28.661739 12348 log.cc:1196] T befc5d75417d47d8b7790017c49fcb71 P 750dbb32b0114697b81aeb7fca556e7f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/befc5d75417d47d8b7790017c49fcb71
I20240502 08:17:28.662729 10083 catalog_manager.cc:4830] TS 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465): tablet befc5d75417d47d8b7790017c49fcb71 (table rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]) successfully deleted
I20240502 08:17:28.674952 12187 raft_consensus.cc:2932] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index -1 to 91, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:28.691340 12148 raft_consensus.cc:1050] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02: attempting to promote NON_VOTER 2fac43dcafe6465b941acc57014c43d3 to VOTER
I20240502 08:17:28.692189 12148 consensus_queue.cc:237] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:28.694164 11535 raft_consensus.cc:1261] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240502 08:17:28.694780 12300 consensus_queue.cc:1035] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 08:17:28.697707 12300 raft_consensus.cc:2932] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 63 to 81, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:28.701604 11535 raft_consensus.cc:2932] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 63 to 81, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:28.703143 10093 catalog_manager.cc:5484] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index 63 to 81, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:28.706087 10237 raft_consensus.cc:2932] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index -1 to 91, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:28.710451 10238 consensus_queue.cc:237] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:28.714355 11535 raft_consensus.cc:1261] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240502 08:17:28.715103 12300 consensus_queue.cc:1035] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 81, Time since last communication: 0.000s
I20240502 08:17:28.719412 10096 catalog_manager.cc:5484] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index -1 to 91, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:28.719497 12300 raft_consensus.cc:2932] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 81 to 83, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:28.723541 11534 raft_consensus.cc:2932] T d426a029b71e4dd4a8852a5cfe708ae0 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 81 to 83, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:28.731237 11800 raft_consensus.cc:1203] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.82->[1.83-1.99]   Dedup: 1.93->[1.94-1.99]
I20240502 08:17:28.736369 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d426a029b71e4dd4a8852a5cfe708ae0 with cas_config_opid_index 81: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:28.738966 10093 catalog_manager.cc:5484] T d426a029b71e4dd4a8852a5cfe708ae0 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index 81 to 83, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:28.744541 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet d426a029b71e4dd4a8852a5cfe708ae0 with delete_type TABLET_DATA_TOMBSTONED (TS f577199118d5411b98b759d6c430daae not found in new config with opid_index 83) from {username='slave'} at 127.0.0.1:52194
I20240502 08:17:28.744926 12313 tablet_replica.cc:330] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: stopping tablet replica
I20240502 08:17:28.745126 12313 raft_consensus.cc:2229] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:28.745698 12313 raft_consensus.cc:2258] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:28.751058 12313 ts_tablet_manager.cc:1875] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:28.780869 12313 ts_tablet_manager.cc:1888] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.79
I20240502 08:17:28.781334 12313 log.cc:1196] T d426a029b71e4dd4a8852a5cfe708ae0 P f577199118d5411b98b759d6c430daae: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/d426a029b71e4dd4a8852a5cfe708ae0
I20240502 08:17:28.783808 10082 catalog_manager.cc:4830] TS f577199118d5411b98b759d6c430daae (127.6.151.66:41685): tablet d426a029b71e4dd4a8852a5cfe708ae0 (table rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]) successfully deleted
I20240502 08:17:28.800612 12233 tablet_bootstrap.cc:492] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=80 overwritten=0 applied=80 ignored=0} inserts{seen=752 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:28.801216 12233 tablet_bootstrap.cc:492] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap complete.
I20240502 08:17:28.801733 12233 ts_tablet_manager.cc:1367] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: Time spent bootstrapping tablet: real 1.465s	user 0.148s	sys 0.024s
I20240502 08:17:28.831561 12233 raft_consensus.cc:348] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:28.835763 12233 raft_consensus.cc:729] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54f96b9fb77540e898821b386cf6d0cb, State: Initialized, Role: LEARNER
I20240502 08:17:28.836581 12233 consensus_queue.cc:260] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.80, Last appended by leader: 80, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:28.849460 12233 ts_tablet_manager.cc:1398] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb: Time spent starting tablet: real 0.042s	user 0.013s	sys 0.000s
I20240502 08:17:28.855299 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session 54f96b9fb77540e898821b386cf6d0cb-fb8df38d1c3340fda6024301f0798487 received from {username='slave'} at 127.6.151.69:59351
I20240502 08:17:28.855489 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session 54f96b9fb77540e898821b386cf6d0cb-fb8df38d1c3340fda6024301f0798487 on tablet fb8df38d1c3340fda6024301f0798487 with peer 54f96b9fb77540e898821b386cf6d0cb
I20240502 08:17:28.871368 12079 raft_consensus.cc:1050] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f: attempting to promote NON_VOTER a57c9c479b194a69b5eb8f9361819a83 to VOTER
I20240502 08:17:28.873494 12079 consensus_queue.cc:237] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:28.878108 10237 raft_consensus.cc:1261] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 08:17:28.879698 12079 consensus_queue.cc:1035] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 08:17:28.880508 11799 raft_consensus.cc:1261] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 08:17:28.881929 12079 consensus_queue.cc:1035] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 08:17:28.884488 12079 raft_consensus.cc:2932] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 83 to 102, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.885880 10235 raft_consensus.cc:2932] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 83 to 102, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.893606 10096 catalog_manager.cc:5484] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 83 to 102, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:28.898398 10499 consensus_queue.cc:237] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:28.902817 11799 raft_consensus.cc:1261] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 08:17:28.903439 12079 consensus_queue.cc:1035] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 08:17:28.908609 11800 raft_consensus.cc:2932] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 83 to 102, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.946691 11670 raft_consensus.cc:1203] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.93]   Dedup: 1.80->[1.81-1.93]
I20240502 08:17:28.987210 12057 raft_consensus.cc:2932] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:28.991358 10083 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82835c215cce4ce49680d56a5bbf0208 with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:29.008879 10096 catalog_manager.cc:5484] T 82835c215cce4ce49680d56a5bbf0208 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 102 to 103, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:29.033780 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet 82835c215cce4ce49680d56a5bbf0208 with delete_type TABLET_DATA_TOMBSTONED (TS 9279bdb860044c90ace75c2f03950e02 not found in new config with opid_index 103) from {username='slave'} at 127.0.0.1:34168
I20240502 08:17:29.034721 11800 raft_consensus.cc:2932] T 82835c215cce4ce49680d56a5bbf0208 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:29.050410 12372 tablet_replica.cc:330] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: stopping tablet replica
I20240502 08:17:29.054833 12372 raft_consensus.cc:2229] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:29.055325 12372 raft_consensus.cc:2258] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:29.055470 12302 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=74 ignored=0} inserts{seen=759 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:29.056042 12302 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: Bootstrap complete.
I20240502 08:17:29.056382 12302 ts_tablet_manager.cc:1367] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: Time spent bootstrapping tablet: real 0.410s	user 0.133s	sys 0.000s
I20240502 08:17:29.057297 12372 ts_tablet_manager.cc:1875] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:29.057408 12302 raft_consensus.cc:348] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:29.060597 12302 raft_consensus.cc:729] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84379f2079ff403eacad196d57abd7ad, State: Initialized, Role: LEARNER
I20240502 08:17:29.060925 12302 consensus_queue.cc:260] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 66 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:29.061671 12302 ts_tablet_manager.cc:1398] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:17:29.063901 12329 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=95 overwritten=0 applied=95 ignored=0} inserts{seen=780 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:29.064383 12329 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap complete.
I20240502 08:17:29.064790 12329 ts_tablet_manager.cc:1367] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: Time spent bootstrapping tablet: real 0.545s	user 0.147s	sys 0.026s
I20240502 08:17:29.065580 12329 raft_consensus.cc:348] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:29.065919 12329 raft_consensus.cc:729] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Initialized, Role: LEARNER
I20240502 08:17:29.066136 12329 consensus_queue.cc:260] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 95, Last appended: 1.95, Last appended by leader: 95, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:29.066829 12329 ts_tablet_manager.cc:1398] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:29.068516 10258 tablet_copy_service.cc:342] P 9279bdb860044c90ace75c2f03950e02: Request end of tablet copy session 2fac43dcafe6465b941acc57014c43d3-00d1cef3b67e4d58aa76c8fd23f2c099 received from {username='slave'} at 127.6.151.68:60407
I20240502 08:17:29.068701 10258 tablet_copy_service.cc:433] P 9279bdb860044c90ace75c2f03950e02: ending tablet copy session 2fac43dcafe6465b941acc57014c43d3-00d1cef3b67e4d58aa76c8fd23f2c099 on tablet 00d1cef3b67e4d58aa76c8fd23f2c099 with peer 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:29.071825 12372 ts_tablet_manager.cc:1888] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.102
I20240502 08:17:29.072023 12372 log.cc:1196] T 82835c215cce4ce49680d56a5bbf0208 P 9279bdb860044c90ace75c2f03950e02: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/82835c215cce4ce49680d56a5bbf0208
I20240502 08:17:29.074621 10082 catalog_manager.cc:4830] TS 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): tablet 82835c215cce4ce49680d56a5bbf0208 (table rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]) successfully deleted
I20240502 08:17:29.076131 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session 84379f2079ff403eacad196d57abd7ad-6fd1419d08a94610b725cc1ed2b04d00 received from {username='slave'} at 127.6.151.71:49841
I20240502 08:17:29.076270 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session 84379f2079ff403eacad196d57abd7ad-6fd1419d08a94610b725cc1ed2b04d00 on tablet 6fd1419d08a94610b725cc1ed2b04d00 with peer 84379f2079ff403eacad196d57abd7ad
I20240502 08:17:29.083801 12233 ts_tablet_manager.cc:902] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:29.110460 10499 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:29.114125 10369 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 08:17:29.117439 12057 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 08:17:29.123271 12080 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:29.128361 10369 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:29.135618 10093 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index -1 to 94, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:29.210314 10435 consensus_peers.cc:485] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f -> Peer 84379f2079ff403eacad196d57abd7ad (127.6.151.71:43045): Couldn't send request to peer 84379f2079ff403eacad196d57abd7ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 138b1526ab364e6bafe0abfa1edb4376. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:29.226502 12354 raft_consensus.cc:1050] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER 54f96b9fb77540e898821b386cf6d0cb to VOTER
I20240502 08:17:29.228163 12354 consensus_queue.cc:237] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.96, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:29.231097 10237 raft_consensus.cc:1261] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240502 08:17:29.235031 12233 tablet_copy_client.cc:308] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:29.236902 11669 raft_consensus.cc:1261] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 97. (index mismatch)
I20240502 08:17:29.237080 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet 75f37a6f3a254fec8b7805185d9ae80f from peer 54f96b9fb77540e898821b386cf6d0cb ({username='slave'} at 127.6.151.69:59351)
I20240502 08:17:29.237236 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet 75f37a6f3a254fec8b7805185d9ae80f from peer 54f96b9fb77540e898821b386cf6d0cb at {username='slave'} at 127.6.151.69:59351: session id = 54f96b9fb77540e898821b386cf6d0cb-75f37a6f3a254fec8b7805185d9ae80f
I20240502 08:17:29.240125 10389 tablet_copy_source_session.cc:205] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:29.241498 12233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75f37a6f3a254fec8b7805185d9ae80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.245052 12081 consensus_queue.cc:1035] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 96, Time since last communication: 0.000s
I20240502 08:17:29.254451 12081 consensus_queue.cc:1035] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 97, Last known committed idx: 94, Time since last communication: 0.000s
I20240502 08:17:29.256469 12187 raft_consensus.cc:2932] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.97: config changed from index 66 to 97, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.267783 11670 raft_consensus.cc:1203] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.95->[1.96-1.98]   Dedup: 1.96->[1.97-1.98]
I20240502 08:17:29.272178 10237 raft_consensus.cc:2932] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 66 to 97, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.272913 12281 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=719 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:29.281237 10096 catalog_manager.cc:5484] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index 66 to 97, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.280829 12233 tablet_copy_client.cc:783] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:29.287012 12233 tablet_copy_client.cc:647] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:29.287837 12281 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap complete.
I20240502 08:17:29.288971 12281 ts_tablet_manager.cc:1367] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: Time spent bootstrapping tablet: real 0.856s	user 0.166s	sys 0.031s
I20240502 08:17:29.289745 12281 raft_consensus.cc:348] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:29.289928 12281 raft_consensus.cc:729] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54f96b9fb77540e898821b386cf6d0cb, State: Initialized, Role: LEARNER
I20240502 08:17:29.290117 12281 consensus_queue.cc:260] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:29.291337 12281 ts_tablet_manager.cc:1398] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:17:29.291409 12233 tablet_copy_client.cc:523] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:29.291967 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session 54f96b9fb77540e898821b386cf6d0cb-2b27b7cc14814dc48c1fe37dbbea9619 received from {username='slave'} at 127.6.151.69:59351
I20240502 08:17:29.292106 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session 54f96b9fb77540e898821b386cf6d0cb-2b27b7cc14814dc48c1fe37dbbea9619 on tablet 2b27b7cc14814dc48c1fe37dbbea9619 with peer 54f96b9fb77540e898821b386cf6d0cb
I20240502 08:17:29.296348 12233 tablet_bootstrap.cc:492] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap starting.
I20240502 08:17:29.302134 12274 tablet_bootstrap.cc:492] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=92 ignored=0} inserts{seen=764 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:29.302618 12274 tablet_bootstrap.cc:492] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: Bootstrap complete.
I20240502 08:17:29.302961 12274 ts_tablet_manager.cc:1367] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: Time spent bootstrapping tablet: real 0.694s	user 0.145s	sys 0.028s
I20240502 08:17:29.304185 12274 raft_consensus.cc:348] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:29.304538 12274 raft_consensus.cc:729] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a57c9c479b194a69b5eb8f9361819a83, State: Initialized, Role: LEARNER
I20240502 08:17:29.304867 12274 consensus_queue.cc:260] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 92, Last appended: 1.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 92 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:29.308899 12274 ts_tablet_manager.cc:1398] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.001s
I20240502 08:17:29.312165 11670 raft_consensus.cc:1203] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.82->[1.83-1.98]   Dedup: 1.88->[1.89-1.98]
I20240502 08:17:29.318676 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session a57c9c479b194a69b5eb8f9361819a83-21631366c38a4a608cb8191903fc535d received from {username='slave'} at 127.6.151.70:50673
I20240502 08:17:29.318858 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session a57c9c479b194a69b5eb8f9361819a83-21631366c38a4a608cb8191903fc535d on tablet 21631366c38a4a608cb8191903fc535d with peer a57c9c479b194a69b5eb8f9361819a83
I20240502 08:17:29.342074 11670 raft_consensus.cc:2932] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.97: config changed from index 66 to 97, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 97 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.358697 12081 raft_consensus.cc:1050] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER 54f96b9fb77540e898821b386cf6d0cb to VOTER
I20240502 08:17:29.359565 12081 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:29.362247 11670 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240502 08:17:29.365022 10497 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240502 08:17:29.366495 10369 consensus_queue.cc:237] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:29.367859 12187 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240502 08:17:29.371021 11670 raft_consensus.cc:1261] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240502 08:17:29.373436 12187 consensus_queue.cc:1035] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240502 08:17:29.373947 12351 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240502 08:17:29.374482 12354 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index 83 to 100, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.383868 10093 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 83 to 100, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:29.385056 10497 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 83 to 100, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.390501 11670 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 83 to 100, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.393208 12028 raft_consensus.cc:2932] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 97 to 99, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.397922 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb8df38d1c3340fda6024301f0798487 with cas_config_opid_index 97: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:29.402187 10369 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:29.404206 11669 raft_consensus.cc:2932] T fb8df38d1c3340fda6024301f0798487 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 97 to 99, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.414232 10096 catalog_manager.cc:5484] T fb8df38d1c3340fda6024301f0798487 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 97 to 99, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:29.420289 11670 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 08:17:29.421089 12355 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 100, Time since last communication: 0.000s
I20240502 08:17:29.424219 10369 consensus_queue.cc:237] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 1.75, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:29.425091 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet fb8df38d1c3340fda6024301f0798487 with delete_type TABLET_DATA_TOMBSTONED (TS 9279bdb860044c90ace75c2f03950e02 not found in new config with opid_index 99) from {username='slave'} at 127.0.0.1:34168
I20240502 08:17:29.425465 12372 tablet_replica.cc:330] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: stopping tablet replica
I20240502 08:17:29.425685 12372 raft_consensus.cc:2229] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:29.426012 12372 raft_consensus.cc:2258] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:29.427558 12372 ts_tablet_manager.cc:1875] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:17:29.427558 10305 consensus_peers.cc:485] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae -> Peer a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70:33091): Couldn't send request to peer a57c9c479b194a69b5eb8f9361819a83. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2c5cbee8e5546739f31b047565e1144. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:29.429952 10497 raft_consensus.cc:1261] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 75. Preceding OpId from leader: term: 1 index: 76. (index mismatch)
I20240502 08:17:29.430763 12081 consensus_queue.cc:1035] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 76, Last known committed idx: 75, Time since last communication: 0.000s
I20240502 08:17:29.433243 12081 raft_consensus.cc:2932] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:29.437662 10499 raft_consensus.cc:2932] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.76: config changed from index -1 to 76, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New config: { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } } }
I20240502 08:17:29.444478 12372 ts_tablet_manager.cc:1888] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.98
I20240502 08:17:29.446619 10093 catalog_manager.cc:5484] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 76, NON_VOTER a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:29.449736 12351 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 100 to 102, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.460891 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b27b7cc14814dc48c1fe37dbbea9619 with cas_config_opid_index 100: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:29.464247 11670 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 100 to 102, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.470404 12372 log.cc:1196] T fb8df38d1c3340fda6024301f0798487 P 9279bdb860044c90ace75c2f03950e02: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/fb8df38d1c3340fda6024301f0798487
I20240502 08:17:29.471593 12233 tablet_bootstrap.cc:492] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=93 overwritten=0 applied=93 ignored=0} inserts{seen=724 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:29.472071 12233 tablet_bootstrap.cc:492] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap complete.
I20240502 08:17:29.472157 10082 catalog_manager.cc:4830] TS 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): tablet fb8df38d1c3340fda6024301f0798487 (table rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]) successfully deleted
I20240502 08:17:29.472414 12233 ts_tablet_manager.cc:1367] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: Time spent bootstrapping tablet: real 0.176s	user 0.127s	sys 0.024s
I20240502 08:17:29.473410 12233 raft_consensus.cc:348] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:29.473760 12233 raft_consensus.cc:729] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54f96b9fb77540e898821b386cf6d0cb, State: Initialized, Role: LEARNER
I20240502 08:17:29.473990 12233 consensus_queue.cc:260] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 93, Last appended: 1.93, Last appended by leader: 93, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:29.475319 12233 ts_tablet_manager.cc:1398] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:17:29.476649 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session 54f96b9fb77540e898821b386cf6d0cb-75f37a6f3a254fec8b7805185d9ae80f received from {username='slave'} at 127.6.151.69:59351
I20240502 08:17:29.476797 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session 54f96b9fb77540e898821b386cf6d0cb-75f37a6f3a254fec8b7805185d9ae80f on tablet 75f37a6f3a254fec8b7805185d9ae80f with peer 54f96b9fb77540e898821b386cf6d0cb
I20240502 08:17:29.485476 10096 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 100 to 102, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:29.488814 11669 raft_consensus.cc:1203] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.90->[1.91-1.96]   Dedup: 1.93->[1.94-1.96]
I20240502 08:17:29.504654 11535 raft_consensus.cc:1203] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.92->[1.93-1.108]   Dedup: 1.95->[1.96-1.108]
I20240502 08:17:29.510140 10479 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b27b7cc14814dc48c1fe37dbbea9619 with delete_type TABLET_DATA_TOMBSTONED (TS 750dbb32b0114697b81aeb7fca556e7f not found in new config with opid_index 102) from {username='slave'} at 127.0.0.1:46754
I20240502 08:17:29.524739 12399 tablet_replica.cc:330] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: stopping tablet replica
I20240502 08:17:29.531440 12399 raft_consensus.cc:2229] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:29.538501 12028 raft_consensus.cc:1050] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER 54f96b9fb77540e898821b386cf6d0cb to VOTER
I20240502 08:17:29.539283 12028 consensus_queue.cc:237] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:29.542029 10499 consensus_queue.cc:237] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:29.544834 11669 raft_consensus.cc:1261] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240502 08:17:29.532312 12399 raft_consensus.cc:2258] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:29.545812 12187 consensus_queue.cc:1035] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240502 08:17:29.547219 10235 raft_consensus.cc:1261] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240502 08:17:29.548409 10235 raft_consensus.cc:1261] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240502 08:17:29.549142 12080 consensus_queue.cc:1035] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
W20240502 08:17:29.551270 10435 consensus_peers.cc:485] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f -> Peer 84379f2079ff403eacad196d57abd7ad (127.6.151.71:43045): Couldn't send request to peer 84379f2079ff403eacad196d57abd7ad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bde4a6ed1250456696a769d280e19ac4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:29.551700 12080 raft_consensus.cc:2932] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:29.552119 10235 raft_consensus.cc:2932] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } } }
I20240502 08:17:29.552829 12187 consensus_queue.cc:1035] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240502 08:17:29.553720 12187 raft_consensus.cc:2932] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 91 to 99, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.558686 12399 ts_tablet_manager.cc:1875] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:29.564702 10093 catalog_manager.cc:5484] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index -1 to 96, NON_VOTER 84379f2079ff403eacad196d57abd7ad (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:29.565354 11669 raft_consensus.cc:2932] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 91 to 99, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.571444 10097 catalog_manager.cc:5484] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 91 to 99, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:29.577639 10237 raft_consensus.cc:2932] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 91 to 99, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.585722 11939 raft_consensus.cc:1203] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.65->[1.66-1.80]   Dedup: 1.75->[1.76-1.80]
I20240502 08:17:29.601068 10369 consensus_queue.cc:237] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:29.603008 11670 raft_consensus.cc:1261] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240502 08:17:29.605219 12404 ts_tablet_manager.cc:902] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: Initiating tablet copy from peer 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:29.605257 12354 consensus_queue.cc:1035] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240502 08:17:29.607739 12354 raft_consensus.cc:2932] T 75f37a6f3a254fec8b7805185d9ae80f P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index 99 to 100, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.610507 12399 ts_tablet_manager.cc:1888] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.100
I20240502 08:17:29.610720 12399 log.cc:1196] T 2b27b7cc14814dc48c1fe37dbbea9619 P 750dbb32b0114697b81aeb7fca556e7f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:29.611197 11669 raft_consensus.cc:2932] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 99 to 100, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.616122 12158 raft_consensus.cc:1050] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: attempting to promote NON_VOTER 2fac43dcafe6465b941acc57014c43d3 to VOTER
I20240502 08:17:29.617075 12158 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:29.618849 10083 catalog_manager.cc:4830] TS 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465): tablet 2b27b7cc14814dc48c1fe37dbbea9619 (table rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]) successfully deleted
I20240502 08:17:29.619398 11533 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240502 08:17:29.621526 12224 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 08:17:29.624315 10093 catalog_manager.cc:5484] T 75f37a6f3a254fec8b7805185d9ae80f P 54f96b9fb77540e898821b386cf6d0cb reported cstate change: config changed from index 99 to 100, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:29.624967 10497 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240502 08:17:29.625866 12224 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 08:17:29.626448 12224 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 93 to 111, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:29.628679 11799 raft_consensus.cc:1203] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.91->[1.92-1.100]   Dedup: 1.92->[1.93-1.100]
I20240502 08:17:29.629139 11533 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 93 to 111, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:29.634105 10497 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 93 to 111, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:29.640849 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet 75f37a6f3a254fec8b7805185d9ae80f with delete_type TABLET_DATA_TOMBSTONED (TS 9279bdb860044c90ace75c2f03950e02 not found in new config with opid_index 100) from {username='slave'} at 127.0.0.1:34168
I20240502 08:17:29.646695 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75f37a6f3a254fec8b7805185d9ae80f with cas_config_opid_index 99: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:29.655454 10092 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 93 to 111, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:29.664649 12372 tablet_replica.cc:330] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: stopping tablet replica
I20240502 08:17:29.664956 12372 raft_consensus.cc:2229] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:29.665609 12372 raft_consensus.cc:2258] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:29.667176 12372 ts_tablet_manager.cc:1875] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:29.688038 12028 raft_consensus.cc:1050] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER a57c9c479b194a69b5eb8f9361819a83 to VOTER
I20240502 08:17:29.688624 12028 consensus_queue.cc:237] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:29.693763 11799 raft_consensus.cc:1261] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 08:17:29.695001 12351 consensus_queue.cc:1035] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 08:17:29.696590 10238 raft_consensus.cc:1261] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 08:17:29.697592 10237 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:29.697564 12354 raft_consensus.cc:2932] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 92 to 103, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:29.700819 11799 raft_consensus.cc:2932] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 92 to 103, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:29.704605 11533 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240502 08:17:29.704562 12187 consensus_queue.cc:1035] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 08:17:29.705211 12412 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 111, Time since last communication: 0.000s
I20240502 08:17:29.712777 11533 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:29.716475 10235 raft_consensus.cc:2932] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 92 to 103, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:29.718503 12158 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:29.748589 10092 catalog_manager.cc:5484] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 92 to 103, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:29.757349 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 with cas_config_opid_index 111: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:29.761986 10092 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index 111 to 113, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:29.766624 10479 tablet_service.cc:1514] Processing DeleteTablet for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 with delete_type TABLET_DATA_TOMBSTONED (TS 750dbb32b0114697b81aeb7fca556e7f not found in new config with opid_index 113) from {username='slave'} at 127.0.0.1:46754
I20240502 08:17:29.767011 12399 tablet_replica.cc:330] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: stopping tablet replica
I20240502 08:17:29.767204 12399 raft_consensus.cc:2229] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:29.767423 12399 raft_consensus.cc:2258] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:29.774354 12399 ts_tablet_manager.cc:1875] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:29.790002 12372 ts_tablet_manager.cc:1888] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.99
I20240502 08:17:29.790163 12372 log.cc:1196] T 75f37a6f3a254fec8b7805185d9ae80f P 9279bdb860044c90ace75c2f03950e02: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/75f37a6f3a254fec8b7805185d9ae80f
I20240502 08:17:29.810211 10082 catalog_manager.cc:4830] TS 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): tablet 75f37a6f3a254fec8b7805185d9ae80f (table rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]) successfully deleted
I20240502 08:17:29.812038 12399 ts_tablet_manager.cc:1888] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.112
I20240502 08:17:29.812377 12399 log.cc:1196] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 750dbb32b0114697b81aeb7fca556e7f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:29.813654 10083 catalog_manager.cc:4830] TS 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465): tablet 00d1cef3b67e4d58aa76c8fd23f2c099 (table rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]) successfully deleted
I20240502 08:17:29.841194 12352 raft_consensus.cc:1050] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER 84379f2079ff403eacad196d57abd7ad to VOTER
I20240502 08:17:29.842013 12352 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:29.845235 10235 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 08:17:29.856413 12081 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 08:17:29.859458 12081 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 66 to 84, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:29.862835 10235 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 66 to 84, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:29.870591 10093 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index 66 to 84, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:29.888265 11939 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240502 08:17:29.889047 12354 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 08:17:29.892109 11938 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 66 to 84, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:29.892612 12404 tablet_copy_client.cc:308] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:38465
I20240502 08:17:29.938622 10369 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:29.940766 10519 tablet_copy_service.cc:140] P 750dbb32b0114697b81aeb7fca556e7f: Received BeginTabletCopySession request for tablet 138b1526ab364e6bafe0abfa1edb4376 from peer 84379f2079ff403eacad196d57abd7ad ({username='slave'} at 127.6.151.71:51215)
I20240502 08:17:29.940955 10519 tablet_copy_service.cc:161] P 750dbb32b0114697b81aeb7fca556e7f: Beginning new tablet copy session on tablet 138b1526ab364e6bafe0abfa1edb4376 from peer 84379f2079ff403eacad196d57abd7ad at {username='slave'} at 127.6.151.71:51215: session id = 84379f2079ff403eacad196d57abd7ad-138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:29.943809 10519 tablet_copy_source_session.cc:205] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:29.943872 11938 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 08:17:29.945075 12028 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 08:17:29.945272 12404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138b1526ab364e6bafe0abfa1edb4376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:29.947649 12081 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:29.951835 11938 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 84 to 87, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:29.957449 10092 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad reported cstate change: config changed from index 84 to 87, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:29.959184 12404 tablet_copy_client.cc:783] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:29.959481 12404 tablet_copy_client.cc:647] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:29.959923 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6fd1419d08a94610b725cc1ed2b04d00 with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:29.967285 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd1419d08a94610b725cc1ed2b04d00 with delete_type TABLET_DATA_TOMBSTONED (TS 9279bdb860044c90ace75c2f03950e02 not found in new config with opid_index 87) from {username='slave'} at 127.0.0.1:34168
I20240502 08:17:29.967686 12372 tablet_replica.cc:330] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: stopping tablet replica
I20240502 08:17:29.967883 12372 raft_consensus.cc:2229] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:29.968125 12372 raft_consensus.cc:2258] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:29.969383 12372 ts_tablet_manager.cc:1875] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:29.980461 12372 ts_tablet_manager.cc:1888] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.86
I20240502 08:17:29.981050 12372 log.cc:1196] T 6fd1419d08a94610b725cc1ed2b04d00 P 9279bdb860044c90ace75c2f03950e02: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:29.982729 10082 catalog_manager.cc:4830] TS 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): tablet 6fd1419d08a94610b725cc1ed2b04d00 (table rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]) successfully deleted
I20240502 08:17:29.991312 12427 ts_tablet_manager.cc:902] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: Initiating tablet copy from peer 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:30.030402 12404 tablet_copy_client.cc:523] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:30.034502 12404 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: Bootstrap starting.
I20240502 08:17:30.065006 12426 ts_tablet_manager.cc:902] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:30.253721 12426 tablet_copy_client.cc:308] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:30.255054 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet a2c5cbee8e5546739f31b047565e1144 from peer a57c9c479b194a69b5eb8f9361819a83 ({username='slave'} at 127.6.151.70:50673)
I20240502 08:17:30.255260 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet a2c5cbee8e5546739f31b047565e1144 from peer a57c9c479b194a69b5eb8f9361819a83 at {username='slave'} at 127.6.151.70:50673: session id = a57c9c479b194a69b5eb8f9361819a83-a2c5cbee8e5546739f31b047565e1144
I20240502 08:17:30.257995 10389 tablet_copy_source_session.cc:205] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:30.260300 12426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c5cbee8e5546739f31b047565e1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.339990 12427 tablet_copy_client.cc:308] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:38465
I20240502 08:17:30.343400 10519 tablet_copy_service.cc:140] P 750dbb32b0114697b81aeb7fca556e7f: Received BeginTabletCopySession request for tablet bde4a6ed1250456696a769d280e19ac4 from peer 84379f2079ff403eacad196d57abd7ad ({username='slave'} at 127.6.151.71:51215)
I20240502 08:17:30.343619 10519 tablet_copy_service.cc:161] P 750dbb32b0114697b81aeb7fca556e7f: Beginning new tablet copy session on tablet bde4a6ed1250456696a769d280e19ac4 from peer 84379f2079ff403eacad196d57abd7ad at {username='slave'} at 127.6.151.71:51215: session id = 84379f2079ff403eacad196d57abd7ad-bde4a6ed1250456696a769d280e19ac4
I20240502 08:17:30.346572 10519 tablet_copy_source_session.cc:205] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:30.348512 12427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4a6ed1250456696a769d280e19ac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:30.356093 12426 tablet_copy_client.cc:783] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:30.356537 12426 tablet_copy_client.cc:647] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:30.362352 12426 tablet_copy_client.cc:523] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:30.365281 12427 tablet_copy_client.cc:783] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:30.365562 12427 tablet_copy_client.cc:647] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:30.377846 12427 tablet_copy_client.cc:523] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:30.381480 12427 tablet_bootstrap.cc:492] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: Bootstrap starting.
I20240502 08:17:30.402663 12426 tablet_bootstrap.cc:492] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap starting.
I20240502 08:17:30.722322 12404 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=105 overwritten=0 applied=104 ignored=0} inserts{seen=789 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:30.732952 12404 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: Bootstrap complete.
I20240502 08:17:30.733407 12404 ts_tablet_manager.cc:1367] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: Time spent bootstrapping tablet: real 0.699s	user 0.167s	sys 0.018s
I20240502 08:17:30.734400 12404 raft_consensus.cc:348] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:30.735114 12404 raft_consensus.cc:729] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84379f2079ff403eacad196d57abd7ad, State: Initialized, Role: LEARNER
I20240502 08:17:30.735420 12404 consensus_queue.cc:260] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 104, Last appended: 1.105, Last appended by leader: 105, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:30.746435 12404 ts_tablet_manager.cc:1398] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.002s
I20240502 08:17:30.750262 10519 tablet_copy_service.cc:342] P 750dbb32b0114697b81aeb7fca556e7f: Request end of tablet copy session 84379f2079ff403eacad196d57abd7ad-138b1526ab364e6bafe0abfa1edb4376 received from {username='slave'} at 127.6.151.71:51215
I20240502 08:17:30.750414 10519 tablet_copy_service.cc:433] P 750dbb32b0114697b81aeb7fca556e7f: ending tablet copy session 84379f2079ff403eacad196d57abd7ad-138b1526ab364e6bafe0abfa1edb4376 on tablet 138b1526ab364e6bafe0abfa1edb4376 with peer 84379f2079ff403eacad196d57abd7ad
I20240502 08:17:30.776155 12426 tablet_bootstrap.cc:492] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=717 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:30.776759 12426 tablet_bootstrap.cc:492] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: Bootstrap complete.
I20240502 08:17:30.777101 12426 ts_tablet_manager.cc:1367] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: Time spent bootstrapping tablet: real 0.375s	user 0.136s	sys 0.030s
I20240502 08:17:30.778527 12426 raft_consensus.cc:348] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:30.778936 12426 raft_consensus.cc:729] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a57c9c479b194a69b5eb8f9361819a83, State: Initialized, Role: LEARNER
I20240502 08:17:30.779213 12426 consensus_queue.cc:260] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 76 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: true } }
I20240502 08:17:30.786671 12426 ts_tablet_manager.cc:1398] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20240502 08:17:30.787695 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session a57c9c479b194a69b5eb8f9361819a83-a2c5cbee8e5546739f31b047565e1144 received from {username='slave'} at 127.6.151.70:50673
I20240502 08:17:30.787820 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session a57c9c479b194a69b5eb8f9361819a83-a2c5cbee8e5546739f31b047565e1144 on tablet a2c5cbee8e5546739f31b047565e1144 with peer a57c9c479b194a69b5eb8f9361819a83
I20240502 08:17:30.799572 12427 tablet_bootstrap.cc:492] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=108 overwritten=0 applied=108 ignored=0} inserts{seen=739 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:30.800143 12427 tablet_bootstrap.cc:492] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: Bootstrap complete.
I20240502 08:17:30.800474 12427 ts_tablet_manager.cc:1367] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: Time spent bootstrapping tablet: real 0.419s	user 0.140s	sys 0.024s
I20240502 08:17:30.801239 12427 raft_consensus.cc:348] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:30.801564 12427 raft_consensus.cc:729] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 84379f2079ff403eacad196d57abd7ad, State: Initialized, Role: LEARNER
I20240502 08:17:30.801785 12427 consensus_queue.cc:260] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 108, Last appended: 1.108, Last appended by leader: 108, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: true } }
I20240502 08:17:30.802507 12427 ts_tablet_manager.cc:1398] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:17:30.805910 10519 tablet_copy_service.cc:342] P 750dbb32b0114697b81aeb7fca556e7f: Request end of tablet copy session 84379f2079ff403eacad196d57abd7ad-bde4a6ed1250456696a769d280e19ac4 received from {username='slave'} at 127.6.151.71:51215
I20240502 08:17:30.806058 10519 tablet_copy_service.cc:433] P 750dbb32b0114697b81aeb7fca556e7f: ending tablet copy session 84379f2079ff403eacad196d57abd7ad-bde4a6ed1250456696a769d280e19ac4 on tablet bde4a6ed1250456696a769d280e19ac4 with peer 84379f2079ff403eacad196d57abd7ad
I20240502 08:17:31.023052 11799 raft_consensus.cc:1203] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.75->[1.76-1.105]   Dedup: 1.86->[1.87-1.105]
I20240502 08:17:31.041050 11936 raft_consensus.cc:1203] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.95->[1.96-1.117]   Dedup: 1.108->[1.109-1.117]
I20240502 08:17:31.078171 12355 raft_consensus.cc:1050] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER a57c9c479b194a69b5eb8f9361819a83 to VOTER
I20240502 08:17:31.081308 12355 consensus_queue.cc:237] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:31.085633 11799 raft_consensus.cc:1261] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 08:17:31.087265 12470 consensus_queue.cc:1035] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 08:17:31.091794 12354 raft_consensus.cc:2932] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 76 to 106, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:31.096664 11799 raft_consensus.cc:2932] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 76 to 106, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:31.106377 10092 catalog_manager.cc:5484] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 reported cstate change: config changed from index 76 to 106, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:31.113727 10497 raft_consensus.cc:1261] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 08:17:31.114856 12028 consensus_queue.cc:1035] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.001s
I20240502 08:17:31.117761 10499 raft_consensus.cc:2932] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 76 to 106, a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:31.167708 11936 raft_consensus.cc:1203] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.93->[1.94-1.126]   Dedup: 1.105->[1.106-1.126]
I20240502 08:17:31.206763 11330 raft_consensus.cc:1050] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f: attempting to promote NON_VOTER 84379f2079ff403eacad196d57abd7ad to VOTER
I20240502 08:17:31.207777 11330 consensus_queue.cc:237] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 117, Committed index: 117, Last appended: 1.118, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:31.212102 10237 raft_consensus.cc:1261] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 08:17:31.212853 12057 consensus_queue.cc:1035] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 117, Time since last communication: 0.000s
I20240502 08:17:31.215341 11938 raft_consensus.cc:1261] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 08:17:31.217864 12080 raft_consensus.cc:2932] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.119: config changed from index 96 to 119, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.221073 10237 raft_consensus.cc:2932] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 96 to 119, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.223771 11330 consensus_queue.cc:1035] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 117, Time since last communication: 0.000s
I20240502 08:17:31.226686 11938 raft_consensus.cc:2932] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 96 to 119, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.285490 10096 catalog_manager.cc:5484] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index 96 to 119, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.297726 12079 raft_consensus.cc:1050] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: attempting to promote NON_VOTER 84379f2079ff403eacad196d57abd7ad to VOTER
I20240502 08:17:31.298552 12079 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 127, Committed index: 127, Last appended: 1.127, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:31.303536 11937 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 1 LEARNER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240502 08:17:31.304345 10369 consensus_queue.cc:237] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:31.314121 11330 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20240502 08:17:31.321760 10369 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 128. (index mismatch)
I20240502 08:17:31.323920 11798 raft_consensus.cc:1261] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240502 08:17:31.328495 12028 consensus_queue.cc:1035] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20240502 08:17:31.332525 12057 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.128: config changed from index 94 to 128, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.334388 12081 raft_consensus.cc:2932] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index 106 to 110, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:31.335613 12079 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 128, Last known committed idx: 127, Time since last communication: 0.000s
I20240502 08:17:31.337672 10499 consensus_queue.cc:237] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:31.340080 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2c5cbee8e5546739f31b047565e1144 with cas_config_opid_index 106: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:31.341526 10091 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 94 to 128, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:31.343415 10092 catalog_manager.cc:5484] T a2c5cbee8e5546739f31b047565e1144 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 106 to 110, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:31.345853 11938 raft_consensus.cc:1261] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240502 08:17:31.347600 11936 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 94 to 128, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.348511 10369 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Committing config change with OpId 1.128: config changed from index 94 to 128, 84379f2079ff403eacad196d57abd7ad (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.351722 10479 tablet_service.cc:1514] Processing DeleteTablet for tablet a2c5cbee8e5546739f31b047565e1144 with delete_type TABLET_DATA_TOMBSTONED (TS 750dbb32b0114697b81aeb7fca556e7f not found in new config with opid_index 110) from {username='slave'} at 127.0.0.1:46754
I20240502 08:17:31.364840 12079 consensus_queue.cc:1035] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 08:17:31.371346 12079 raft_consensus.cc:2932] T bde4a6ed1250456696a769d280e19ac4 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 119 to 123, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.356817 11798 raft_consensus.cc:2932] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 106 to 110, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:31.378392 10083 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bde4a6ed1250456696a769d280e19ac4 with cas_config_opid_index 119: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:31.385150 11936 raft_consensus.cc:2932] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 119 to 123, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.394338 10092 catalog_manager.cc:5484] T bde4a6ed1250456696a769d280e19ac4 P 84379f2079ff403eacad196d57abd7ad reported cstate change: config changed from index 119 to 123, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:31.407043 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet bde4a6ed1250456696a769d280e19ac4 with delete_type TABLET_DATA_TOMBSTONED (TS 9279bdb860044c90ace75c2f03950e02 not found in new config with opid_index 123) from {username='slave'} at 127.0.0.1:34168
I20240502 08:17:31.415450 12502 tablet_replica.cc:330] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: stopping tablet replica
I20240502 08:17:31.418839 12502 raft_consensus.cc:2229] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:31.419332 12502 raft_consensus.cc:2258] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:31.422881 12502 ts_tablet_manager.cc:1875] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:31.435173 12502 ts_tablet_manager.cc:1888] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.109
I20240502 08:17:31.443781 12502 log.cc:1196] T a2c5cbee8e5546739f31b047565e1144 P 750dbb32b0114697b81aeb7fca556e7f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/a2c5cbee8e5546739f31b047565e1144
I20240502 08:17:31.445495 10083 catalog_manager.cc:4830] TS 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465): tablet a2c5cbee8e5546739f31b047565e1144 (table rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]) successfully deleted
I20240502 08:17:31.507771 12506 tablet_replica.cc:330] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: stopping tablet replica
I20240502 08:17:31.508492 12506 raft_consensus.cc:2229] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:31.508749 12506 pending_rounds.cc:62] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Trying to abort 1 pending ops.
I20240502 08:17:31.508873 12506 pending_rounds.cc:68] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Aborting op as it isn't in flight: id { term: 1 index: 122 } timestamp: 7023156638933188608 op_type: WRITE_OP write_request { tablet_id: "bde4a6ed1250456696a769d280e19ac4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\346\240s\01393\024\334\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""1830e29e62dbfef4" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023156638806929408 authz_token { token_data: "\010\277\232\315\261\006\"3\n\005slave\022*\n ee2d386900364032847f6f2be742b075\020\001\030\001 \001(\001" signature: "<redacted>""\245\331\006\326,\322cN\273\320\330\211\3552\327T=%\207\262;\351\264\226+F\375{\361i\2647V>3\237\365I\305F\305<ML@\"\210\003\224\301\347CU6\311jo\327\276JI\220\255\353" signing_key_seq_num: 0 } } request_id { client_id: "28478e5e460e4f0f92c7ae6dce33405e" seq_no: 381 first_incomplete_seq_no: 381 attempt_no: 0 }
I20240502 08:17:31.514396 12506 raft_consensus.cc:2258] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:31.515445 12506 ts_tablet_manager.cc:1875] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:31.536151 12506 ts_tablet_manager.cc:1888] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.122
I20240502 08:17:31.539347 12506 log.cc:1196] T bde4a6ed1250456696a769d280e19ac4 P 9279bdb860044c90ace75c2f03950e02: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/bde4a6ed1250456696a769d280e19ac4
I20240502 08:17:31.540675 10082 catalog_manager.cc:4830] TS 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): tablet bde4a6ed1250456696a769d280e19ac4 (table rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]) successfully deleted
I20240502 08:17:32.082918 10369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "21631366c38a4a608cb8191903fc535d"
dest_uuid: "f577199118d5411b98b759d6c430daae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54636
I20240502 08:17:32.083312 10369 raft_consensus.cc:595] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Received request to transfer leadership
W20240502 08:17:32.083885 10326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:32.096311 10326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
I20240502 08:17:32.098800 12081 raft_consensus.cc:979] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: : Instructing follower a57c9c479b194a69b5eb8f9361819a83 to start an election
I20240502 08:17:32.099069 12081 raft_consensus.cc:1067] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Signalling peer a57c9c479b194a69b5eb8f9361819a83 to start an election
I20240502 08:17:32.102304 11800 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "21631366c38a4a608cb8191903fc535d"
dest_uuid: "a57c9c479b194a69b5eb8f9361819a83"
 from {username='slave'} at 127.6.151.66:51925
I20240502 08:17:32.102710 11800 raft_consensus.cc:482] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:32.102972 11800 raft_consensus.cc:3037] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:32.108285 11800 raft_consensus.cc:504] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
W20240502 08:17:32.121163 10326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:32.126596 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
I20240502 08:17:32.139372 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21631366c38a4a608cb8191903fc535d" candidate_uuid: "a57c9c479b194a69b5eb8f9361819a83" candidate_term: 2 candidate_status { last_received { term: 1 index: 126 } } ignore_live_leader: true dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:32.139716 10369 raft_consensus.cc:3032] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:32.139850 10369 raft_consensus.cc:729] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:32.140252 10369 consensus_queue.cc:260] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 126, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:32.140805 10369 raft_consensus.cc:3037] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:32.145406 10369 raft_consensus.cc:2454] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57c9c479b194a69b5eb8f9361819a83 in term 2.
I20240502 08:17:32.151791 11734 leader_election.cc:302] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a57c9c479b194a69b5eb8f9361819a83, f577199118d5411b98b759d6c430daae; no voters: 
W20240502 08:17:32.165124 10326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Illegal state: replica f577199118d5411b98b759d6c430daae is not leader of this config: current role FOLLOWER
I20240502 08:17:32.199090 11800 leader_election.cc:288] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [CANDIDATE]: Term 2 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685), 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:32.201297 12553 raft_consensus.cc:2781] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 2 FOLLOWER]: Leader election won for term 2
W20240502 08:17:32.209648 11734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.151.66:41685, user_credentials={real_user=slave}} blocked reactor thread for 57900.1us
I20240502 08:17:32.243234 12553 raft_consensus.cc:686] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 2 LEADER]: Becoming Leader. State: Replica: a57c9c479b194a69b5eb8f9361819a83, State: Running, Role: LEADER
I20240502 08:17:32.244208 12553 consensus_queue.cc:237] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 126, Committed index: 126, Last appended: 1.126, Last appended by leader: 126, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
W20240502 08:17:32.252180 10198 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55156: Illegal state: replica 9279bdb860044c90ace75c2f03950e02 is not leader of this config: current role FOLLOWER
W20240502 08:17:32.275204 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Illegal state: replica f577199118d5411b98b759d6c430daae is not leader of this config: current role FOLLOWER
W20240502 08:17:32.288509 10198 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55156: Illegal state: replica 9279bdb860044c90ace75c2f03950e02 is not leader of this config: current role FOLLOWER
I20240502 08:17:32.304471 10091 catalog_manager.cc:5484] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 reported cstate change: term changed from 1 to 2, leader changed from f577199118d5411b98b759d6c430daae (127.6.151.66) to a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70). New cstate: current_term: 2 leader_uuid: "a57c9c479b194a69b5eb8f9361819a83" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:32.368489 10237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21631366c38a4a608cb8191903fc535d" candidate_uuid: "a57c9c479b194a69b5eb8f9361819a83" candidate_term: 2 candidate_status { last_received { term: 1 index: 126 } } ignore_live_leader: true dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:32.368753 10237 raft_consensus.cc:3037] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:32.373559 10369 raft_consensus.cc:1261] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer a57c9c479b194a69b5eb8f9361819a83: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 2 index: 128. (index mismatch)
I20240502 08:17:32.382030 12553 consensus_queue.cc:1035] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240502 08:17:32.395242 10237 raft_consensus.cc:2454] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a57c9c479b194a69b5eb8f9361819a83 in term 2.
I20240502 08:17:32.395941 10238 raft_consensus.cc:1261] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Refusing update from remote peer a57c9c479b194a69b5eb8f9361819a83: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 2 index: 128. (index mismatch)
I20240502 08:17:32.401484 12556 consensus_queue.cc:1035] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 126, Time since last communication: 0.000s
I20240502 08:17:32.412387 12507 mvcc.cc:205] Tried to move back new op lower bound from 7023156643293794304 to 7023156642798739456. Current Snapshot: MvccSnapshot[applied={T|T < 7023156643293794304 or (T in {7023156643293794304})}]
W20240502 08:17:32.472826 11734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43271, user_credentials={real_user=slave}} blocked reactor thread for 60848.7us
I20240502 08:17:32.491468 10497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "138b1526ab364e6bafe0abfa1edb4376"
dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43642
I20240502 08:17:32.491779 10497 raft_consensus.cc:595] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:32.501798 11800 consensus_queue.cc:237] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 2.129, Last appended by leader: 126, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
W20240502 08:17:32.501878 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.508296 10238 raft_consensus.cc:1261] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Refusing update from remote peer a57c9c479b194a69b5eb8f9361819a83: Log matching property violated. Preceding OpId in replica: term: 2 index: 129. Preceding OpId from leader: term: 2 index: 130. (index mismatch)
W20240502 08:17:32.509210 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.510602 12556 consensus_queue.cc:1035] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 08:17:32.513947 12556 raft_consensus.cc:2932] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 [term 2 LEADER]: Committing config change with OpId 2.130: config changed from index 103 to 130, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:32.518823 10238 raft_consensus.cc:2932] T 21631366c38a4a608cb8191903fc535d P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Committing config change with OpId 2.130: config changed from index 103 to 130, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } }
I20240502 08:17:32.522352 10084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21631366c38a4a608cb8191903fc535d with cas_config_opid_index 103: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 08:17:32.525506 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.525610 10092 catalog_manager.cc:5484] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 reported cstate change: config changed from index 103 to 130, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "a57c9c479b194a69b5eb8f9361819a83" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:32.535236 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.543205 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.550210 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet 21631366c38a4a608cb8191903fc535d with delete_type TABLET_DATA_TOMBSTONED (TS f577199118d5411b98b759d6c430daae not found in new config with opid_index 130) from {username='slave'} at 127.0.0.1:52194
W20240502 08:17:32.552029 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.557667 12590 tablet_replica.cc:330] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: stopping tablet replica
I20240502 08:17:32.557896 12590 raft_consensus.cc:2229] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:32.558203 12590 raft_consensus.cc:2258] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 08:17:32.562613 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.569499 12590 ts_tablet_manager.cc:1875] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:17:32.574939 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.585902 12590 ts_tablet_manager.cc:1888] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.129
I20240502 08:17:32.587163 12590 log.cc:1196] T 21631366c38a4a608cb8191903fc535d P f577199118d5411b98b759d6c430daae: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/21631366c38a4a608cb8191903fc535d
I20240502 08:17:32.588375 10082 catalog_manager.cc:4830] TS f577199118d5411b98b759d6c430daae (127.6.151.66:41685): tablet 21631366c38a4a608cb8191903fc535d (table rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]) successfully deleted
W20240502 08:17:32.588770 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.604476 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.617122 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.633052 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.649893 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.668449 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.685886 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.704785 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.727231 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.731693 10497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "138b1526ab364e6bafe0abfa1edb4376"
dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:43668
I20240502 08:17:32.731967 10497 raft_consensus.cc:595] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Received request to transfer leadership
W20240502 08:17:32.746693 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.770972 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.795820 10459 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.823197 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.850569 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
W20240502 08:17:32.879164 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Service unavailable: leader transfer in progress
I20240502 08:17:32.883765 11330 raft_consensus.cc:979] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: : Instructing follower 84379f2079ff403eacad196d57abd7ad to start an election
I20240502 08:17:32.883955 11330 raft_consensus.cc:1067] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Signalling peer 84379f2079ff403eacad196d57abd7ad to start an election
I20240502 08:17:32.885517 11938 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "138b1526ab364e6bafe0abfa1edb4376"
dest_uuid: "84379f2079ff403eacad196d57abd7ad"
 from {username='slave'} at 127.6.151.67:40217
I20240502 08:17:32.885890 11938 raft_consensus.cc:482] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:32.886157 11938 raft_consensus.cc:3037] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:32.889362 11938 raft_consensus.cc:504] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:32.891774 11938 leader_election.cc:288] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [CANDIDATE]: Term 2 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685), 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:32.895104 10497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138b1526ab364e6bafe0abfa1edb4376" candidate_uuid: "84379f2079ff403eacad196d57abd7ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 148 } } ignore_live_leader: true dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:32.895356 10497 raft_consensus.cc:3032] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:32.895455 10497 raft_consensus.cc:729] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Running, Role: LEADER
I20240502 08:17:32.895799 10497 consensus_queue.cc:260] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 1.148, Last appended by leader: 148, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:32.896281 10497 raft_consensus.cc:3037] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:32.899701 10497 raft_consensus.cc:2454] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84379f2079ff403eacad196d57abd7ad in term 2.
I20240502 08:17:32.900117 10369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138b1526ab364e6bafe0abfa1edb4376" candidate_uuid: "84379f2079ff403eacad196d57abd7ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 148 } } ignore_live_leader: true dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:32.900343 10369 raft_consensus.cc:3037] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:32.900955 11874 leader_election.cc:302] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, 84379f2079ff403eacad196d57abd7ad; no voters: 
I20240502 08:17:32.901660 12607 raft_consensus.cc:2781] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:32.903921 10369 raft_consensus.cc:2454] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84379f2079ff403eacad196d57abd7ad in term 2.
W20240502 08:17:32.910593 10458 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42380: Illegal state: replica 750dbb32b0114697b81aeb7fca556e7f is not leader of this config: current role FOLLOWER
I20240502 08:17:32.927735 12607 raft_consensus.cc:686] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 2 LEADER]: Becoming Leader. State: Replica: 84379f2079ff403eacad196d57abd7ad, State: Running, Role: LEADER
I20240502 08:17:32.932495 12607 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 1.148, Last appended by leader: 148, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:32.940080 10091 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad reported cstate change: term changed from 1 to 2, leader changed from 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) to 84379f2079ff403eacad196d57abd7ad (127.6.151.71). New cstate: current_term: 2 leader_uuid: "84379f2079ff403eacad196d57abd7ad" committed_config { opid_index: 128 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:32.945739 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46642: Illegal state: replica f577199118d5411b98b759d6c430daae is not leader of this config: current role FOLLOWER
I20240502 08:17:33.040983 10497 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 2 index: 150. (index mismatch)
I20240502 08:17:33.042023 12608 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I20240502 08:17:33.049718 10369 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 148. Preceding OpId from leader: term: 2 index: 150. (index mismatch)
I20240502 08:17:33.050577 12608 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 149, Last known committed idx: 148, Time since last communication: 0.000s
I20240502 08:17:33.066181 12565 mvcc.cc:205] Tried to move back new op lower bound from 7023156646041911296 to 7023156645616939008. Current Snapshot: MvccSnapshot[applied={T|T < 7023156646041911296 or (T in {7023156646041911296})}]
I20240502 08:17:33.112021 11938 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 2.150, Last appended by leader: 148, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:33.126067 10369 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 150. Preceding OpId from leader: term: 2 index: 151. (index mismatch)
I20240502 08:17:33.126710 12608 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240502 08:17:33.132002 12608 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 2 LEADER]: Committing config change with OpId 2.151: config changed from index 128 to 151, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:33.133972 10369 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Committing config change with OpId 2.151: config changed from index 128 to 151, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } }
I20240502 08:17:33.138546 10084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 138b1526ab364e6bafe0abfa1edb4376 with cas_config_opid_index 128: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:33.140094 10091 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad reported cstate change: config changed from index 128 to 151, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New cstate: current_term: 2 leader_uuid: "84379f2079ff403eacad196d57abd7ad" committed_config { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:33.145964 10479 tablet_service.cc:1514] Processing DeleteTablet for tablet 138b1526ab364e6bafe0abfa1edb4376 with delete_type TABLET_DATA_TOMBSTONED (TS 750dbb32b0114697b81aeb7fca556e7f not found in new config with opid_index 151) from {username='slave'} at 127.0.0.1:46754
I20240502 08:17:33.160542 12623 tablet_replica.cc:330] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: stopping tablet replica
I20240502 08:17:33.160780 12623 raft_consensus.cc:2229] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:33.161037 12623 raft_consensus.cc:2258] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:33.164681 12623 ts_tablet_manager.cc:1875] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:33.174531 12623 ts_tablet_manager.cc:1888] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.150
I20240502 08:17:33.174804 12623 log.cc:1196] T 138b1526ab364e6bafe0abfa1edb4376 P 750dbb32b0114697b81aeb7fca556e7f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:33.175896 10083 catalog_manager.cc:4830] TS 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465): tablet 138b1526ab364e6bafe0abfa1edb4376 (table rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]) successfully deleted
I20240502 08:17:34.080734 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:34.129323 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:34.151294 10479 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:34.161520 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:34.163426 11919 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:34.172354 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:34.330984 10348 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:17:34.421380 10174 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.71:43045, user_credentials={real_user=slave}} blocked reactor thread for 80469.7us
I20240502 08:17:34.863190 11939 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 2.185, Last appended by leader: 148, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:34.909996 10369 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 185. Preceding OpId from leader: term: 2 index: 186. (index mismatch)
I20240502 08:17:34.916422 12608 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 185, Time since last communication: 0.000s
I20240502 08:17:34.921389 12607 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 2 LEADER]: Committing config change with OpId 2.186: config changed from index 151 to 186, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:34.928901 10369 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Committing config change with OpId 2.186: config changed from index 151 to 186, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:34.937528 10096 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad reported cstate change: config changed from index 151 to 186, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New cstate: current_term: 2 leader_uuid: "84379f2079ff403eacad196d57abd7ad" committed_config { opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:34.948441 11873 consensus_peers.cc:485] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad -> Peer 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733): Couldn't send request to peer 54f96b9fb77540e898821b386cf6d0cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 138b1526ab364e6bafe0abfa1edb4376. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:35.112214 10369 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 169, Committed index: 169, Last appended: 1.169, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:35.123706 11939 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 169. Preceding OpId from leader: term: 1 index: 170. (index mismatch)
I20240502 08:17:35.124444 12688 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 170, Last known committed idx: 169, Time since last communication: 0.000s
I20240502 08:17:35.127919 12538 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.170: config changed from index 87 to 170, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } } }
I20240502 08:17:35.129077 11938 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.170: config changed from index 87 to 170, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New config: { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } } }
I20240502 08:17:35.140734 10096 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 87 to 170, NON_VOTER 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:35.153825 10302 consensus_peers.cc:485] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae -> Peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Couldn't send request to peer 2fac43dcafe6465b941acc57014c43d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fd1419d08a94610b725cc1ed2b04d00. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:35.192394 10368 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 158, Committed index: 158, Last appended: 1.158, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
W20240502 08:17:35.194841 10303 consensus_peers.cc:485] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae -> Peer 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733): Couldn't send request to peer 54f96b9fb77540e898821b386cf6d0cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b02fadb91da44d33bd8f45847433a1ab. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:35.195160 10497 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 158. Preceding OpId from leader: term: 1 index: 160. (index mismatch)
I20240502 08:17:35.196126 12688 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 159, Last known committed idx: 158, Time since last communication: 0.000s
I20240502 08:17:35.201299 12688 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.159: config changed from index -1 to 159, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:35.202612 10497 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.159: config changed from index -1 to 159, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New config: { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } } }
I20240502 08:17:35.212234 10092 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index -1 to 159, NON_VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:35.335834 12731 ts_tablet_manager.cc:902] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: Initiating tablet copy from peer 84379f2079ff403eacad196d57abd7ad (127.6.151.71:43045)
I20240502 08:17:35.465003 12731 tablet_copy_client.cc:308] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.71:43045
I20240502 08:17:35.528978 11959 tablet_copy_service.cc:140] P 84379f2079ff403eacad196d57abd7ad: Received BeginTabletCopySession request for tablet 138b1526ab364e6bafe0abfa1edb4376 from peer 54f96b9fb77540e898821b386cf6d0cb ({username='slave'} at 127.6.151.69:35469)
I20240502 08:17:35.529292 11959 tablet_copy_service.cc:161] P 84379f2079ff403eacad196d57abd7ad: Beginning new tablet copy session on tablet 138b1526ab364e6bafe0abfa1edb4376 from peer 54f96b9fb77540e898821b386cf6d0cb at {username='slave'} at 127.6.151.69:35469: session id = 54f96b9fb77540e898821b386cf6d0cb-138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:35.532939 11959 tablet_copy_source_session.cc:205] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:35.543877 12731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138b1526ab364e6bafe0abfa1edb4376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.552639 12731 tablet_copy_client.cc:783] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:35.553175 12731 tablet_copy_client.cc:647] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:35.569518 12731 tablet_copy_client.cc:523] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:35.573559 12731 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap starting.
I20240502 08:17:35.653332 12751 ts_tablet_manager.cc:902] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:35.742364 12751 tablet_copy_client.cc:308] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:35.820408 12761 ts_tablet_manager.cc:902] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:35.887348 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet 6fd1419d08a94610b725cc1ed2b04d00 from peer 2fac43dcafe6465b941acc57014c43d3 ({username='slave'} at 127.6.151.68:38705)
I20240502 08:17:35.887557 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet 6fd1419d08a94610b725cc1ed2b04d00 from peer 2fac43dcafe6465b941acc57014c43d3 at {username='slave'} at 127.6.151.68:38705: session id = 2fac43dcafe6465b941acc57014c43d3-6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:35.890280 10389 tablet_copy_source_session.cc:205] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:35.891846 12751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd1419d08a94610b725cc1ed2b04d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:35.901973 12751 tablet_copy_client.cc:783] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:35.902333 12751 tablet_copy_client.cc:647] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:35.904736 12751 tablet_copy_client.cc:523] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:35.927093 12751 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap starting.
I20240502 08:17:36.014861 12761 tablet_copy_client.cc:308] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:36.015591 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet b02fadb91da44d33bd8f45847433a1ab from peer 54f96b9fb77540e898821b386cf6d0cb ({username='slave'} at 127.6.151.69:59351)
I20240502 08:17:36.015779 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet b02fadb91da44d33bd8f45847433a1ab from peer 54f96b9fb77540e898821b386cf6d0cb at {username='slave'} at 127.6.151.69:59351: session id = 54f96b9fb77540e898821b386cf6d0cb-b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:36.018616 10389 tablet_copy_source_session.cc:205] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:36.019815 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02fadb91da44d33bd8f45847433a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:36.036253 12761 tablet_copy_client.cc:783] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:36.036967 12761 tablet_copy_client.cc:647] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:36.045303 12761 tablet_copy_client.cc:523] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:36.049423 12761 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap starting.
W20240502 08:17:36.160092 10435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:33091, user_credentials={real_user=slave}} blocked reactor thread for 75815.5us
I20240502 08:17:36.231262 12731 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=202 overwritten=0 applied=201 ignored=0} inserts{seen=882 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:17:36.231997 12731 tablet_bootstrap.cc:492] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap complete.
I20240502 08:17:36.232436 12731 ts_tablet_manager.cc:1367] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: Time spent bootstrapping tablet: real 0.659s	user 0.249s	sys 0.040s
I20240502 08:17:36.233669 12731 raft_consensus.cc:348] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [term 2 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:36.234441 12731 raft_consensus.cc:729] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 54f96b9fb77540e898821b386cf6d0cb, State: Initialized, Role: LEARNER
I20240502 08:17:36.234748 12731 consensus_queue.cc:260] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 201, Last appended: 2.202, Last appended by leader: 202, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 186 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:36.250705 12731 ts_tablet_manager.cc:1398] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.004s
I20240502 08:17:36.251716 11959 tablet_copy_service.cc:342] P 84379f2079ff403eacad196d57abd7ad: Request end of tablet copy session 54f96b9fb77540e898821b386cf6d0cb-138b1526ab364e6bafe0abfa1edb4376 received from {username='slave'} at 127.6.151.69:35469
I20240502 08:17:36.251919 11959 tablet_copy_service.cc:433] P 84379f2079ff403eacad196d57abd7ad: ending tablet copy session 54f96b9fb77540e898821b386cf6d0cb-138b1526ab364e6bafe0abfa1edb4376 on tablet 138b1526ab364e6bafe0abfa1edb4376 with peer 54f96b9fb77540e898821b386cf6d0cb
I20240502 08:17:36.528457 12751 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=183 overwritten=0 applied=183 ignored=0} inserts{seen=865 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:36.529151 12751 tablet_bootstrap.cc:492] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: Bootstrap complete.
I20240502 08:17:36.530464 12751 ts_tablet_manager.cc:1367] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: Time spent bootstrapping tablet: real 0.603s	user 0.262s	sys 0.017s
I20240502 08:17:36.532644 12751 raft_consensus.cc:348] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:36.535571 12751 raft_consensus.cc:729] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Initialized, Role: LEARNER
I20240502 08:17:36.535969 12751 consensus_queue.cc:260] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 183, Last appended: 1.183, Last appended by leader: 183, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 170 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: true } }
I20240502 08:17:36.541546 12751 ts_tablet_manager.cc:1398] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 08:17:36.543151 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session 2fac43dcafe6465b941acc57014c43d3-6fd1419d08a94610b725cc1ed2b04d00 received from {username='slave'} at 127.6.151.68:38705
I20240502 08:17:36.543292 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session 2fac43dcafe6465b941acc57014c43d3-6fd1419d08a94610b725cc1ed2b04d00 on tablet 6fd1419d08a94610b725cc1ed2b04d00 with peer 2fac43dcafe6465b941acc57014c43d3
I20240502 08:17:36.559458 11668 raft_consensus.cc:1203] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.185->[2.186-2.213]   Dedup: 2.202->[2.203-2.213]
I20240502 08:17:36.642771 11534 raft_consensus.cc:1203] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.169->[1.170-1.196]   Dedup: 1.183->[1.184-1.196]
I20240502 08:17:36.670495 12607 raft_consensus.cc:1050] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad: attempting to promote NON_VOTER 54f96b9fb77540e898821b386cf6d0cb to VOTER
I20240502 08:17:36.671361 12607 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 2.214, Last appended by leader: 148, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:36.673908 11668 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [term 2 LEARNER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20240502 08:17:36.674469 10368 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 2 index: 215. (index mismatch)
I20240502 08:17:36.674523 12703 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 213, Time since last communication: 0.000s
I20240502 08:17:36.675750 12703 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 213, Time since last communication: 0.000s
I20240502 08:17:36.679631 12608 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 2 LEADER]: Committing config change with OpId 2.215: config changed from index 186 to 215, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:36.687316 11668 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 186 to 215, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:36.687345 10368 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Committing config change with OpId 2.215: config changed from index 186 to 215, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:36.693249 10092 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb reported cstate change: config changed from index 186 to 215, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "84379f2079ff403eacad196d57abd7ad" committed_config { opid_index: 215 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:36.808724 12538 raft_consensus.cc:1050] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER 2fac43dcafe6465b941acc57014c43d3 to VOTER
I20240502 08:17:36.809612 12538 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 1.198, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:36.812224 11939 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 199. (index mismatch)
I20240502 08:17:36.813261 12081 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240502 08:17:36.813683 11534 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 199. (index mismatch)
I20240502 08:17:36.814383 12081 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s
I20240502 08:17:36.816159 12081 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.199: config changed from index 170 to 199, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:36.817795 11534 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Committing config change with OpId 1.199: config changed from index 170 to 199, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:36.819252 11939 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Committing config change with OpId 1.199: config changed from index 170 to 199, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:36.893384 10097 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 reported cstate change: config changed from index 170 to 199, 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:36.976713 12761 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=173 overwritten=0 applied=173 ignored=0} inserts{seen=858 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:36.977193 12761 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Bootstrap complete.
I20240502 08:17:36.977556 12761 ts_tablet_manager.cc:1367] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Time spent bootstrapping tablet: real 0.928s	user 0.261s	sys 0.013s
I20240502 08:17:36.978324 12761 raft_consensus.cc:348] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:36.978559 12761 raft_consensus.cc:729] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54f96b9fb77540e898821b386cf6d0cb, State: Initialized, Role: LEARNER
I20240502 08:17:36.978806 12761 consensus_queue.cc:260] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 173, Last appended: 1.173, Last appended by leader: 173, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 159 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: true } }
I20240502 08:17:36.987965 12761 ts_tablet_manager.cc:1398] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Time spent starting tablet: real 0.010s	user 0.000s	sys 0.003s
I20240502 08:17:36.989068 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session 54f96b9fb77540e898821b386cf6d0cb-b02fadb91da44d33bd8f45847433a1ab received from {username='slave'} at 127.6.151.69:59351
I20240502 08:17:36.989216 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session 54f96b9fb77540e898821b386cf6d0cb-b02fadb91da44d33bd8f45847433a1ab on tablet b02fadb91da44d33bd8f45847433a1ab with peer 54f96b9fb77540e898821b386cf6d0cb
I20240502 08:17:37.070695 11939 consensus_queue.cc:237] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 222, Committed index: 222, Last appended: 2.222, Last appended by leader: 148, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:37.074031 11668 raft_consensus.cc:1261] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 222. Preceding OpId from leader: term: 2 index: 223. (index mismatch)
I20240502 08:17:37.074764 12608 consensus_queue.cc:1035] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 222, Time since last communication: 0.000s
I20240502 08:17:37.077425 12608 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad [term 2 LEADER]: Committing config change with OpId 2.223: config changed from index 215 to 223, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:37.081809 11668 raft_consensus.cc:2932] T 138b1526ab364e6bafe0abfa1edb4376 P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Committing config change with OpId 2.223: config changed from index 215 to 223, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:37.085806 10084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 138b1526ab364e6bafe0abfa1edb4376 with cas_config_opid_index 215: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:37.087680 10091 catalog_manager.cc:5484] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad reported cstate change: config changed from index 215 to 223, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "84379f2079ff403eacad196d57abd7ad" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:37.098542 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet 138b1526ab364e6bafe0abfa1edb4376 with delete_type TABLET_DATA_TOMBSTONED (TS f577199118d5411b98b759d6c430daae not found in new config with opid_index 223) from {username='slave'} at 127.0.0.1:52194
I20240502 08:17:37.132817 12817 tablet_replica.cc:330] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: stopping tablet replica
I20240502 08:17:37.139885 12817 raft_consensus.cc:2229] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:37.143154 12817 raft_consensus.cc:2258] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:37.148715 12817 ts_tablet_manager.cc:1875] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:37.168816 12817 ts_tablet_manager.cc:1888] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.222
I20240502 08:17:37.176229 12817 log.cc:1196] T 138b1526ab364e6bafe0abfa1edb4376 P f577199118d5411b98b759d6c430daae: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:37.180933 10082 catalog_manager.cc:4830] TS f577199118d5411b98b759d6c430daae (127.6.151.66:41685): tablet 138b1526ab364e6bafe0abfa1edb4376 (table rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]) successfully deleted
I20240502 08:17:37.200894 11667 raft_consensus.cc:1203] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.158->[1.159-1.196]   Dedup: 1.173->[1.174-1.196]
I20240502 08:17:37.441622 10368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6fd1419d08a94610b725cc1ed2b04d00"
dest_uuid: "f577199118d5411b98b759d6c430daae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54738
I20240502 08:17:37.441861 10368 raft_consensus.cc:595] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Received request to transfer leadership
W20240502 08:17:37.465229 10326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
I20240502 08:17:37.473965 12814 raft_consensus.cc:1050] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER 54f96b9fb77540e898821b386cf6d0cb to VOTER
I20240502 08:17:37.475311 12814 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 1.201, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:37.478432 10497 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 1 index: 202. (index mismatch)
I20240502 08:17:37.479095 12470 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I20240502 08:17:37.483332 11667 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 201. Preceding OpId from leader: term: 1 index: 202. (index mismatch)
I20240502 08:17:37.484848 12470 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.202: config changed from index 159 to 202, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
W20240502 08:17:37.486761 10326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
I20240502 08:17:37.487216 12814 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 202, Last known committed idx: 201, Time since last communication: 0.000s
I20240502 08:17:37.487753 10497 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Committing config change with OpId 1.202: config changed from index 159 to 202, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:37.490615 11667 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.202: config changed from index 159 to 202, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:37.497063 10096 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 159 to 202, 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
W20240502 08:17:37.523841 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.538690 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.554458 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.574209 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.588338 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.610997 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.628860 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.670806 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.694788 10326 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.717954 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.736917 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.753934 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.773311 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
W20240502 08:17:37.799579 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Service unavailable: leader transfer in progress
I20240502 08:17:37.832757 12538 raft_consensus.cc:979] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: : Instructing follower 84379f2079ff403eacad196d57abd7ad to start an election
I20240502 08:17:37.832943 12538 raft_consensus.cc:1067] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Signalling peer 84379f2079ff403eacad196d57abd7ad to start an election
I20240502 08:17:37.833669 11936 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6fd1419d08a94610b725cc1ed2b04d00"
dest_uuid: "84379f2079ff403eacad196d57abd7ad"
 from {username='slave'} at 127.6.151.66:39609
I20240502 08:17:37.833871 11936 raft_consensus.cc:482] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:37.833981 11936 raft_consensus.cc:3037] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:37.836848 11936 raft_consensus.cc:504] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:37.847440 11936 leader_election.cc:288] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [CANDIDATE]: Term 2 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685), 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265)
I20240502 08:17:37.856130 10368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd1419d08a94610b725cc1ed2b04d00" candidate_uuid: "84379f2079ff403eacad196d57abd7ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 210 } } ignore_live_leader: true dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:37.856369 10368 raft_consensus.cc:3032] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:37.856472 10368 raft_consensus.cc:729] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:37.856729 10368 consensus_queue.cc:260] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 1.210, Last appended by leader: 210, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:37.857115 10368 raft_consensus.cc:3037] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:17:37.859474 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46622: Illegal state: replica f577199118d5411b98b759d6c430daae is not leader of this config: current role FOLLOWER
I20240502 08:17:37.868070 10368 raft_consensus.cc:2454] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84379f2079ff403eacad196d57abd7ad in term 2.
I20240502 08:17:37.868669 11873 leader_election.cc:302] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84379f2079ff403eacad196d57abd7ad, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:37.869000 12703 raft_consensus.cc:2781] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:37.869139 12703 raft_consensus.cc:686] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 2 LEADER]: Becoming Leader. State: Replica: 84379f2079ff403eacad196d57abd7ad, State: Running, Role: LEADER
I20240502 08:17:37.869483 12703 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 210, Committed index: 210, Last appended: 1.210, Last appended by leader: 210, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:37.874785 10096 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad reported cstate change: term changed from 1 to 2, leader changed from f577199118d5411b98b759d6c430daae (127.6.151.66) to 84379f2079ff403eacad196d57abd7ad (127.6.151.71). New cstate: current_term: 2 leader_uuid: "84379f2079ff403eacad196d57abd7ad" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:37.886967 10368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b02fadb91da44d33bd8f45847433a1ab"
dest_uuid: "f577199118d5411b98b759d6c430daae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54754
I20240502 08:17:37.887231 10368 raft_consensus.cc:595] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:37.895171 11533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd1419d08a94610b725cc1ed2b04d00" candidate_uuid: "84379f2079ff403eacad196d57abd7ad" candidate_term: 2 candidate_status { last_received { term: 1 index: 210 } } ignore_live_leader: true dest_uuid: "2fac43dcafe6465b941acc57014c43d3"
I20240502 08:17:37.895417 11533 raft_consensus.cc:3037] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:17:37.896382 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
I20240502 08:17:37.898319 11533 raft_consensus.cc:2454] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84379f2079ff403eacad196d57abd7ad in term 2.
W20240502 08:17:37.904603 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:37.912824 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:37.923617 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:37.932521 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:37.939101 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:37.950124 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
I20240502 08:17:37.958884 11533 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 210. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
I20240502 08:17:37.959272 10368 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 210. Preceding OpId from leader: term: 2 index: 212. (index mismatch)
I20240502 08:17:37.959944 12608 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
I20240502 08:17:37.960444 12703 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 211, Last known committed idx: 210, Time since last communication: 0.000s
W20240502 08:17:37.963625 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
I20240502 08:17:37.970942 11936 consensus_queue.cc:237] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 212, Committed index: 212, Last appended: 2.212, Last appended by leader: 210, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } }
I20240502 08:17:37.973323 11533 raft_consensus.cc:1261] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Refusing update from remote peer 84379f2079ff403eacad196d57abd7ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 212. Preceding OpId from leader: term: 2 index: 213. (index mismatch)
I20240502 08:17:37.973907 12703 consensus_queue.cc:1035] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 213, Last known committed idx: 212, Time since last communication: 0.000s
I20240502 08:17:37.976982 12703 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad [term 2 LEADER]: Committing config change with OpId 2.213: config changed from index 199 to 213, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:37.980931 11533 raft_consensus.cc:2932] T 6fd1419d08a94610b725cc1ed2b04d00 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Committing config change with OpId 2.213: config changed from index 199 to 213, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } }
I20240502 08:17:37.985890 10084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6fd1419d08a94610b725cc1ed2b04d00 with cas_config_opid_index 199: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:37.986914 10091 catalog_manager.cc:5484] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad reported cstate change: config changed from index 199 to 213, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "84379f2079ff403eacad196d57abd7ad" committed_config { opid_index: 213 OBSOLETE_local: false peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:17:37.989053 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
I20240502 08:17:37.991858 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd1419d08a94610b725cc1ed2b04d00 with delete_type TABLET_DATA_TOMBSTONED (TS f577199118d5411b98b759d6c430daae not found in new config with opid_index 213) from {username='slave'} at 127.0.0.1:52194
I20240502 08:17:37.995833 12875 tablet_replica.cc:330] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: stopping tablet replica
I20240502 08:17:37.996946 12875 raft_consensus.cc:2229] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:37.997414 12875 pending_rounds.cc:62] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Trying to abort 2 pending ops.
I20240502 08:17:37.997593 12875 pending_rounds.cc:68] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Aborting op as it isn't in flight: id { term: 2 index: 212 } timestamp: 7023156666193850368 op_type: WRITE_OP write_request { tablet_id: "6fd1419d08a94610b725cc1ed2b04d00" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\273\340\241u\361\n\032\006\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""eda12f4c62507bf6" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023156663992045568 authz_token { token_data: "\010\277\232\315\261\006\"3\n\005slave\022*\n 5c45765ce98b49a080ae7d35c2170fdb\020\001\030\001 \001(\001" signature: "<redacted>""\213\330\337?\237I\226\233\005\026\032\301\332ky\r\246M[\277)A\321\372\263Ae\266\331\357\257\275;\000\251G\314b \320\201\307\351?\326\010\257\316\306\227\033\027*,y\004H\323\323\331\241pLp" signing_key_seq_num: 0 } } request_id { client_id: "c73ec2fa1c7f4270be2ba781fd9930ec" seq_no: 639 first_incomplete_seq_no: 639 attempt_no: 18 }
I20240502 08:17:37.998459 12875 pending_rounds.cc:68] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Aborting op as it isn't in flight: id { term: 2 index: 211 } timestamp: 7023156665836568576 op_type: NO_OP noop_request { }
I20240502 08:17:37.998874 12875 raft_consensus.cc:2866] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 08:17:37.999112 12875 raft_consensus.cc:2258] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:38.001690 12875 ts_tablet_manager.cc:1875] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:17:38.008975 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
I20240502 08:17:38.023295 12875 ts_tablet_manager.cc:1888] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.212
I20240502 08:17:38.023473 12875 log.cc:1196] T 6fd1419d08a94610b725cc1ed2b04d00 P f577199118d5411b98b759d6c430daae: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:38.024566 10082 catalog_manager.cc:4830] TS f577199118d5411b98b759d6c430daae (127.6.151.66:41685): tablet 6fd1419d08a94610b725cc1ed2b04d00 (table rebalance_test_table_0 [id=5c45765ce98b49a080ae7d35c2170fdb]) successfully deleted
W20240502 08:17:38.025866 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.043006 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.058951 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.073741 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.091339 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.113162 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.131803 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.152544 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
I20240502 08:17:38.169749 10368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b02fadb91da44d33bd8f45847433a1ab"
dest_uuid: "f577199118d5411b98b759d6c430daae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:54766
I20240502 08:17:38.169983 10368 raft_consensus.cc:595] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Received request to transfer leadership
W20240502 08:17:38.177151 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.202131 10328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
W20240502 08:17:38.231159 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Service unavailable: leader transfer in progress
I20240502 08:17:38.231670 12687 raft_consensus.cc:979] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: : Instructing follower 750dbb32b0114697b81aeb7fca556e7f to start an election
I20240502 08:17:38.231876 12687 raft_consensus.cc:1067] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Signalling peer 750dbb32b0114697b81aeb7fca556e7f to start an election
I20240502 08:17:38.233103 10499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b02fadb91da44d33bd8f45847433a1ab"
dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
 from {username='slave'} at 127.6.151.66:59159
I20240502 08:17:38.233337 10499 raft_consensus.cc:482] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:38.233475 10499 raft_consensus.cc:3037] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.235973 10499 raft_consensus.cc:504] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:38.237941 10499 leader_election.cc:288] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685), 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733)
I20240502 08:17:38.238952 10368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02fadb91da44d33bd8f45847433a1ab" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 218 } } ignore_live_leader: true dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:38.239174 10368 raft_consensus.cc:3032] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:38.239269 10368 raft_consensus.cc:729] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:38.239539 10368 consensus_queue.cc:260] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:38.239931 10368 raft_consensus.cc:3037] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.244594 10368 raft_consensus.cc:2454] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 2.
I20240502 08:17:38.246873 10433 leader_election.cc:302] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:38.247370 12837 raft_consensus.cc:2781] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:38.247510 12837 raft_consensus.cc:686] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 2 LEADER]: Becoming Leader. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Running, Role: LEADER
I20240502 08:17:38.247875 12837 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:38.251929 10091 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: term changed from 1 to 2, leader changed from f577199118d5411b98b759d6c430daae (127.6.151.66) to 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67). New cstate: current_term: 2 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 202 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:17:38.257284 10327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46636: Illegal state: replica f577199118d5411b98b759d6c430daae is not leader of this config: current role FOLLOWER
I20240502 08:17:38.271556 11668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b02fadb91da44d33bd8f45847433a1ab" candidate_uuid: "750dbb32b0114697b81aeb7fca556e7f" candidate_term: 2 candidate_status { last_received { term: 1 index: 218 } } ignore_live_leader: true dest_uuid: "54f96b9fb77540e898821b386cf6d0cb"
I20240502 08:17:38.271908 11668 raft_consensus.cc:3037] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:38.275409 11668 raft_consensus.cc:2454] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 750dbb32b0114697b81aeb7fca556e7f in term 2.
I20240502 08:17:38.332383 11668 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240502 08:17:38.332530 10368 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 2 index: 220. (index mismatch)
I20240502 08:17:38.333611 12712 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240502 08:17:38.334203 12712 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240502 08:17:38.344652 12684 mvcc.cc:205] Tried to move back new op lower bound from 7023156667720945664 to 7023156667387006976. Current Snapshot: MvccSnapshot[applied={T|T < 7023156667720945664 or (T in {7023156667720945664})}]
I20240502 08:17:38.353227 10499 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 2.220, Last appended by leader: 218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } }
I20240502 08:17:38.355664 11670 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 220. Preceding OpId from leader: term: 2 index: 221. (index mismatch)
I20240502 08:17:38.360152 12836 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s
I20240502 08:17:38.363569 12836 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 2 LEADER]: Committing config change with OpId 2.221: config changed from index 202 to 221, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:38.364272 11668 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Committing config change with OpId 2.221: config changed from index 202 to 221, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } }
I20240502 08:17:38.368911 10083 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b02fadb91da44d33bd8f45847433a1ab with cas_config_opid_index 202: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:38.373800 10091 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 202 to 221, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:38.382107 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet b02fadb91da44d33bd8f45847433a1ab with delete_type TABLET_DATA_TOMBSTONED (TS f577199118d5411b98b759d6c430daae not found in new config with opid_index 221) from {username='slave'} at 127.0.0.1:52194
I20240502 08:17:38.382982 12875 tablet_replica.cc:330] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: stopping tablet replica
I20240502 08:17:38.383275 12875 raft_consensus.cc:2229] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:38.383555 12875 raft_consensus.cc:2258] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:38.385066 12875 ts_tablet_manager.cc:1875] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:38.403791 12875 ts_tablet_manager.cc:1888] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.220
I20240502 08:17:38.404884 12875 log.cc:1196] T b02fadb91da44d33bd8f45847433a1ab P f577199118d5411b98b759d6c430daae: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:38.406158 10082 catalog_manager.cc:4830] TS f577199118d5411b98b759d6c430daae (127.6.151.66:41685): tablet b02fadb91da44d33bd8f45847433a1ab (table rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]) successfully deleted
I20240502 08:17:39.680480 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:39.690228 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:39.699126 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:39.804863 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:39.845686 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:40.057758 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:40.099503 10348 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:17:40.534126 10433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:44733, user_credentials={real_user=slave}} blocked reactor thread for 57252.5us
I20240502 08:17:40.786470 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:40.831116 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:40.918789 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:40.922302 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:41.010744 10348 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:41.156817 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:41.174268 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:43.209769 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:43.216670 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:43.235563 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:43.283337 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:17:43.342842 10305 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:33091, user_credentials={real_user=slave}} blocked reactor thread for 47277.5us
I20240502 08:17:43.403479 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:43.403951 10348 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:43.416673 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:44.567163 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:44.648567 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:44.655457 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:44.678438 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:44.695941 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:44.719144 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:44.785624 10348 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:17:44.813225 11734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43271, user_credentials={real_user=slave}} blocked reactor thread for 87993.7us
I20240502 08:17:45.551429 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:45.563784 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:45.577451 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:45.579861 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:45.580499 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:45.598022 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:45.619413 10348 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 108c4a65ead24022acd2eaae8cc9aece | 127.6.151.126:36085 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:35619 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 2fac43dcafe6465b941acc57014c43d3 | 127.6.151.68:37265 | HEALTHY | <none>   |       1        |       0
 54f96b9fb77540e898821b386cf6d0cb | 127.6.151.69:44733 | HEALTHY | <none>   |       0        |       0
 750dbb32b0114697b81aeb7fca556e7f | 127.6.151.67:38465 | HEALTHY | <none>   |       4        |       0
 84379f2079ff403eacad196d57abd7ad | 127.6.151.71:43045 | HEALTHY | <none>   |       2        |       0
 9279bdb860044c90ace75c2f03950e02 | 127.6.151.65:43271 | HEALTHY | <none>   |       3        |       0
 a57c9c479b194a69b5eb8f9361819a83 | 127.6.151.70:33091 | HEALTHY | <none>   |       1        |       0
 f577199118d5411b98b759d6c430daae | 127.6.151.66:41685 | HEALTHY | <none>   |       5        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       7

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 7 server(s) checked
 local_ip_for_outbound_sockets            | 127.6.151.65                                                                                                                                                    | experimental          | 127.6.151.65:43271
 local_ip_for_outbound_sockets            | 127.6.151.66                                                                                                                                                    | experimental          | 127.6.151.66:41685
 local_ip_for_outbound_sockets            | 127.6.151.67                                                                                                                                                    | experimental          | 127.6.151.67:38465
 local_ip_for_outbound_sockets            | 127.6.151.68                                                                                                                                                    | experimental          | 127.6.151.68:37265
 local_ip_for_outbound_sockets            | 127.6.151.69                                                                                                                                                    | experimental          | 127.6.151.69:44733
 local_ip_for_outbound_sockets            | 127.6.151.70                                                                                                                                                    | experimental          | 127.6.151.70:33091
 local_ip_for_outbound_sockets            | 127.6.151.71                                                                                                                                                    | experimental          | 127.6.151.71:43045
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 7 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 7 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 7 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 7 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 7 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.6.151.65:43271
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.6.151.66:41685
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.6.151.67:38465
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.6.151.68:37265
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.6.151.69:44733
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.6.151.70:33091
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.6.151.71:43045

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:35619 | all 7 server(s) checked
 time_source         | builtin            | all 7 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 8 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 5
 Third Quartile | 5
 Maximum        | 5

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 7
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 08:17:45.675276  6749 log_verifier.cc:126] Checking tablet 00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:45.831346  6749 log_verifier.cc:177] Verified matching terms for 415 ops in tablet 00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:45.831887  6749 log_verifier.cc:126] Checking tablet 138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:45.990240  6749 log_verifier.cc:177] Verified matching terms for 358 ops in tablet 138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:45.990783  6749 log_verifier.cc:126] Checking tablet 21631366c38a4a608cb8191903fc535d
I20240502 08:17:46.126749  6749 log_verifier.cc:177] Verified matching terms for 350 ops in tablet 21631366c38a4a608cb8191903fc535d
I20240502 08:17:46.127267  6749 log_verifier.cc:126] Checking tablet 2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:46.291890  6749 log_verifier.cc:177] Verified matching terms for 397 ops in tablet 2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:46.292407  6749 log_verifier.cc:126] Checking tablet 6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:46.415974  6749 log_verifier.cc:177] Verified matching terms for 324 ops in tablet 6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:46.416433  6749 log_verifier.cc:126] Checking tablet 75f37a6f3a254fec8b7805185d9ae80f
I20240502 08:17:46.551586  6749 log_verifier.cc:177] Verified matching terms for 341 ops in tablet 75f37a6f3a254fec8b7805185d9ae80f
I20240502 08:17:46.552109  6749 log_verifier.cc:126] Checking tablet 82835c215cce4ce49680d56a5bbf0208
I20240502 08:17:46.703771  6749 log_verifier.cc:177] Verified matching terms for 401 ops in tablet 82835c215cce4ce49680d56a5bbf0208
I20240502 08:17:46.704295  6749 log_verifier.cc:126] Checking tablet a2c5cbee8e5546739f31b047565e1144
I20240502 08:17:46.843039  6749 log_verifier.cc:177] Verified matching terms for 362 ops in tablet a2c5cbee8e5546739f31b047565e1144
I20240502 08:17:46.854771  6749 log_verifier.cc:126] Checking tablet b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:46.981891  6749 log_verifier.cc:177] Verified matching terms for 320 ops in tablet b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:46.982339  6749 log_verifier.cc:126] Checking tablet bde4a6ed1250456696a769d280e19ac4
I20240502 08:17:47.122727  6749 log_verifier.cc:177] Verified matching terms for 373 ops in tablet bde4a6ed1250456696a769d280e19ac4
I20240502 08:17:47.123204  6749 log_verifier.cc:126] Checking tablet befc5d75417d47d8b7790017c49fcb71
I20240502 08:17:47.304279  6749 log_verifier.cc:177] Verified matching terms for 431 ops in tablet befc5d75417d47d8b7790017c49fcb71
I20240502 08:17:47.304857  6749 log_verifier.cc:126] Checking tablet c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:47.457830  6749 log_verifier.cc:177] Verified matching terms for 384 ops in tablet c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:47.458330  6749 log_verifier.cc:126] Checking tablet d426a029b71e4dd4a8852a5cfe708ae0
I20240502 08:17:47.598562  6749 log_verifier.cc:177] Verified matching terms for 348 ops in tablet d426a029b71e4dd4a8852a5cfe708ae0
I20240502 08:17:47.599381  6749 log_verifier.cc:126] Checking tablet d471c5fb3045495aa1ec0a774e6164d8
I20240502 08:17:47.729334  6749 log_verifier.cc:177] Verified matching terms for 332 ops in tablet d471c5fb3045495aa1ec0a774e6164d8
I20240502 08:17:47.729771  6749 log_verifier.cc:126] Checking tablet f5f8034ce87a4c85a72c53ba07f178d1
I20240502 08:17:47.746490  6749 log_verifier.cc:177] Verified matching terms for 2 ops in tablet f5f8034ce87a4c85a72c53ba07f178d1
I20240502 08:17:47.746583  6749 log_verifier.cc:126] Checking tablet fb8df38d1c3340fda6024301f0798487
I20240502 08:17:47.894457  6749 log_verifier.cc:177] Verified matching terms for 380 ops in tablet fb8df38d1c3340fda6024301f0798487
I20240502 08:17:47.897742  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.72:0
--local_ip_for_outbound_sockets=127.6.151.72
--webserver_interface=127.6.151.72
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--builtin_ntp_servers=127.6.151.84:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:48.278303 13224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:48.278611 13224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:48.278777 13224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:48.286378 13224 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:48.286509 13224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:48.286669 13224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.72
I20240502 08:17:48.296052 13224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.151.72
--webserver_port=0
--tserver_master_addrs=127.6.151.126:36085
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.72
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:48.297370 13224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:48.299242 13224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:48.311285 13231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:48.311553 13233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:48.312251 13224 server_base.cc:1034] running on GCE node
W20240502 08:17:48.312361 13230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:48.313068 13224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:48.314116 13224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:48.315318 13224 hybrid_clock.cc:648] HybridClock initialized: now 1714637868315257 us; error 52 us; skew 500 ppm
I20240502 08:17:48.315676 13224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:48.326447 13224 webserver.cc:460] Webserver started at http://127.6.151.72:34435/ using document root <none> and password file <none>
I20240502 08:17:48.327327 13224 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:48.327445 13224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:48.327771 13224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:48.330996 13224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ca969bc50f72467b94d8436d37be2f8e"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-x3qn"
I20240502 08:17:48.331943 13224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ca969bc50f72467b94d8436d37be2f8e"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-x3qn"
I20240502 08:17:48.337884 13224 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20240502 08:17:48.341895 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:48.343377 13224 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:17:48.343518 13224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ca969bc50f72467b94d8436d37be2f8e"
format_stamp: "Formatted at 2024-05-02 08:17:48 on dist-test-slave-x3qn"
I20240502 08:17:48.343710 13224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:48.364658 13224 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:48.365561 13224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:48.365857 13224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:48.366943 13224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:48.368463 13224 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:48.368549 13224 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:48.368655 13224 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:48.368752 13224 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:48.402976 13352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.72:34005 every 8 connection(s)
I20240502 08:17:48.403064 13224 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.72:34005
I20240502 08:17:48.404855 13224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:17:48.411754  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 13224
I20240502 08:17:48.411960  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:17:48.420961 13353 heartbeater.cc:347] Connected to a master server at 127.6.151.126:36085
I20240502 08:17:48.421289 13353 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:48.422140 13353 heartbeater.cc:511] Master 127.6.151.126:36085 requested a full tablet report, sending...
I20240502 08:17:48.423621 10096 ts_manager.cc:194] Registered new tserver with Master: ca969bc50f72467b94d8436d37be2f8e (127.6.151.72:34005)
I20240502 08:17:48.424742 10096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.72:43389
I20240502 08:17:48.961491 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:48.981613 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:48.985005 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:48.985584 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:49.027647 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:49.032012 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:49.065961 13288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:49.067224 10348 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:49.267097 11515 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:49.277974 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:49.286525 10479 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:49.298486 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:49.300437 11650 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:49.319743 13288 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:17:49.327164 10217 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:49.327836 10348 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:17:49.425880 10499 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 384, Committed index: 384, Last appended: 1.384, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:49.428089 13353 heartbeater.cc:503] Master 127.6.151.126:36085 was elected leader, sending a full tablet report...
I20240502 08:17:49.430051 11799 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 384. Preceding OpId from leader: term: 1 index: 385. (index mismatch)
I20240502 08:17:49.430809 13024 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 385, Last known committed idx: 384, Time since last communication: 0.000s
I20240502 08:17:49.433979 13396 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.385: config changed from index 90 to 385, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
I20240502 08:17:49.434697 11799 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.385: config changed from index 90 to 385, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
W20240502 08:17:49.436440 10435 consensus_peers.cc:485] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f -> Peer ca969bc50f72467b94d8436d37be2f8e (127.6.151.72:34005): Couldn't send request to peer ca969bc50f72467b94d8436d37be2f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0a4038683ac48449ccad5eaeeef5694. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:49.439783 10096 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 90 to 385, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:49.453012 10235 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 415, Committed index: 415, Last appended: 1.415, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 416 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:49.455682 11533 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 415. Preceding OpId from leader: term: 1 index: 416. (index mismatch)
I20240502 08:17:49.456321 13026 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 416, Last known committed idx: 415, Time since last communication: 0.000s
I20240502 08:17:49.460166 13026 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.416: config changed from index 113 to 416, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 416 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
I20240502 08:17:49.460673 11533 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Committing config change with OpId 1.416: config changed from index 113 to 416, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 416 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
W20240502 08:17:49.463618 10174 consensus_peers.cc:485] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 -> Peer ca969bc50f72467b94d8436d37be2f8e (127.6.151.72:34005): Couldn't send request to peer ca969bc50f72467b94d8436d37be2f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00d1cef3b67e4d58aa76c8fd23f2c099. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:49.466133 10096 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 reported cstate change: config changed from index 113 to 416, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 416 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
I20240502 08:17:49.492031 10368 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 397, Committed index: 397, Last appended: 1.397, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:49.494944 11667 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 397. Preceding OpId from leader: term: 1 index: 398. (index mismatch)
I20240502 08:17:49.495523 13473 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 398, Last known committed idx: 397, Time since last communication: 0.000s
I20240502 08:17:49.498399 13163 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.398: config changed from index 102 to 398, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
I20240502 08:17:49.499102 11667 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.398: config changed from index 102 to 398, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
W20240502 08:17:49.501075 10305 consensus_peers.cc:485] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae -> Peer ca969bc50f72467b94d8436d37be2f8e (127.6.151.72:34005): Couldn't send request to peer ca969bc50f72467b94d8436d37be2f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b27b7cc14814dc48c1fe37dbbea9619. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:49.504926 10097 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb reported cstate change: config changed from index 102 to 398, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
I20240502 08:17:49.515934 10499 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 320, Committed index: 320, Last appended: 2.320, Last appended by leader: 218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:49.517940 11667 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 320. Preceding OpId from leader: term: 2 index: 321. (index mismatch)
W20240502 08:17:49.518074 10435 consensus_peers.cc:485] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f -> Peer ca969bc50f72467b94d8436d37be2f8e (127.6.151.72:34005): Couldn't send request to peer ca969bc50f72467b94d8436d37be2f8e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b02fadb91da44d33bd8f45847433a1ab. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:49.518429 13024 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 321, Last known committed idx: 320, Time since last communication: 0.000s
I20240502 08:17:49.520860 13396 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 2 LEADER]: Committing config change with OpId 2.321: config changed from index 221 to 321, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
I20240502 08:17:49.521370 11667 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Committing config change with OpId 2.321: config changed from index 221 to 321, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New config: { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } } }
I20240502 08:17:49.525774 10096 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 221 to 321, NON_VOTER ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) added. New cstate: current_term: 2 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:49.827476 13524 ts_tablet_manager.cc:902] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: Initiating tablet copy from peer 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:49.829414 13524 tablet_copy_client.cc:308] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:38465
I20240502 08:17:49.836007 10519 tablet_copy_service.cc:140] P 750dbb32b0114697b81aeb7fca556e7f: Received BeginTabletCopySession request for tablet c0a4038683ac48449ccad5eaeeef5694 from peer ca969bc50f72467b94d8436d37be2f8e ({username='slave'} at 127.6.151.72:57809)
I20240502 08:17:49.836185 10519 tablet_copy_service.cc:161] P 750dbb32b0114697b81aeb7fca556e7f: Beginning new tablet copy session on tablet c0a4038683ac48449ccad5eaeeef5694 from peer ca969bc50f72467b94d8436d37be2f8e at {username='slave'} at 127.6.151.72:57809: session id = ca969bc50f72467b94d8436d37be2f8e-c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:49.838270 10519 tablet_copy_source_session.cc:205] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:49.842512 13524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a4038683ac48449ccad5eaeeef5694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.852274 13524 tablet_copy_client.cc:783] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:49.852573 13524 tablet_copy_client.cc:647] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:49.855947 13524 tablet_copy_client.cc:523] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:49.859392 13524 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap starting.
I20240502 08:17:49.911295 13524 log.cc:824] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:49.928699 13531 ts_tablet_manager.cc:902] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: Initiating tablet copy from peer 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465)
I20240502 08:17:49.939888 13531 tablet_copy_client.cc:308] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:38465
I20240502 08:17:49.940877 10519 tablet_copy_service.cc:140] P 750dbb32b0114697b81aeb7fca556e7f: Received BeginTabletCopySession request for tablet b02fadb91da44d33bd8f45847433a1ab from peer ca969bc50f72467b94d8436d37be2f8e ({username='slave'} at 127.6.151.72:57809)
I20240502 08:17:49.941110 10519 tablet_copy_service.cc:161] P 750dbb32b0114697b81aeb7fca556e7f: Beginning new tablet copy session on tablet b02fadb91da44d33bd8f45847433a1ab from peer ca969bc50f72467b94d8436d37be2f8e at {username='slave'} at 127.6.151.72:57809: session id = ca969bc50f72467b94d8436d37be2f8e-b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:49.944181 10519 tablet_copy_source_session.cc:205] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:49.945657 13531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b02fadb91da44d33bd8f45847433a1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:49.951335 13531 tablet_copy_client.cc:783] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:49.951583 13531 tablet_copy_client.cc:647] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:49.954383 13531 tablet_copy_client.cc:523] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:49.958262 13531 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: Bootstrap starting.
I20240502 08:17:50.045955 13550 ts_tablet_manager.cc:902] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: Initiating tablet copy from peer f577199118d5411b98b759d6c430daae (127.6.151.66:41685)
I20240502 08:17:50.055714 13550 tablet_copy_client.cc:308] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:41685
I20240502 08:17:50.066533 10389 tablet_copy_service.cc:140] P f577199118d5411b98b759d6c430daae: Received BeginTabletCopySession request for tablet 2b27b7cc14814dc48c1fe37dbbea9619 from peer ca969bc50f72467b94d8436d37be2f8e ({username='slave'} at 127.6.151.72:37271)
I20240502 08:17:50.066777 10389 tablet_copy_service.cc:161] P f577199118d5411b98b759d6c430daae: Beginning new tablet copy session on tablet 2b27b7cc14814dc48c1fe37dbbea9619 from peer ca969bc50f72467b94d8436d37be2f8e at {username='slave'} at 127.6.151.72:37271: session id = ca969bc50f72467b94d8436d37be2f8e-2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:50.069032 10389 tablet_copy_source_session.cc:205] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:50.070747 13550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b27b7cc14814dc48c1fe37dbbea9619. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.077869 13550 tablet_copy_client.cc:783] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:50.078320 13550 tablet_copy_client.cc:647] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:50.082082 13550 tablet_copy_client.cc:523] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:50.086953 13550 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap starting.
I20240502 08:17:50.087215 13558 ts_tablet_manager.cc:902] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: Initiating tablet copy from peer 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
I20240502 08:17:50.131068 13558 tablet_copy_client.cc:308] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43271
I20240502 08:17:50.172688 10258 tablet_copy_service.cc:140] P 9279bdb860044c90ace75c2f03950e02: Received BeginTabletCopySession request for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 from peer ca969bc50f72467b94d8436d37be2f8e ({username='slave'} at 127.6.151.72:58491)
I20240502 08:17:50.172930 10258 tablet_copy_service.cc:161] P 9279bdb860044c90ace75c2f03950e02: Beginning new tablet copy session on tablet 00d1cef3b67e4d58aa76c8fd23f2c099 from peer ca969bc50f72467b94d8436d37be2f8e at {username='slave'} at 127.6.151.72:58491: session id = ca969bc50f72467b94d8436d37be2f8e-00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:50.175944 10258 tablet_copy_source_session.cc:205] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:17:50.177699 13558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00d1cef3b67e4d58aa76c8fd23f2c099. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:50.184516 13558 tablet_copy_client.cc:783] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 0 data blocks...
I20240502 08:17:50.184805 13558 tablet_copy_client.cc:647] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:17:50.188482 13558 tablet_copy_client.cc:523] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:17:50.192299 13558 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap starting.
I20240502 08:17:50.421514 13531 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=321 overwritten=0 applied=321 ignored=0} inserts{seen=1002 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:50.432564 13531 tablet_bootstrap.cc:492] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: Bootstrap complete.
I20240502 08:17:50.433394 13531 ts_tablet_manager.cc:1367] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: Time spent bootstrapping tablet: real 0.475s	user 0.400s	sys 0.034s
I20240502 08:17:50.441082 13531 raft_consensus.cc:348] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.451339 13531 raft_consensus.cc:729] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ca969bc50f72467b94d8436d37be2f8e, State: Initialized, Role: LEARNER
I20240502 08:17:50.452335 13531 consensus_queue.cc:260] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 321, Last appended: 2.321, Last appended by leader: 321, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 321 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.458853 13531 ts_tablet_manager.cc:1398] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.005s
I20240502 08:17:50.463246 10519 tablet_copy_service.cc:342] P 750dbb32b0114697b81aeb7fca556e7f: Request end of tablet copy session ca969bc50f72467b94d8436d37be2f8e-b02fadb91da44d33bd8f45847433a1ab received from {username='slave'} at 127.6.151.72:57809
I20240502 08:17:50.463397 10519 tablet_copy_service.cc:433] P 750dbb32b0114697b81aeb7fca556e7f: ending tablet copy session ca969bc50f72467b94d8436d37be2f8e-b02fadb91da44d33bd8f45847433a1ab on tablet b02fadb91da44d33bd8f45847433a1ab with peer ca969bc50f72467b94d8436d37be2f8e
I20240502 08:17:50.470892 13524 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=385 overwritten=0 applied=385 ignored=0} inserts{seen=1013 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:50.471521 13524 tablet_bootstrap.cc:492] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap complete.
I20240502 08:17:50.471910 13524 ts_tablet_manager.cc:1367] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: Time spent bootstrapping tablet: real 0.613s	user 0.430s	sys 0.058s
I20240502 08:17:50.473016 13524 raft_consensus.cc:348] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.473351 13524 raft_consensus.cc:729] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca969bc50f72467b94d8436d37be2f8e, State: Initialized, Role: LEARNER
I20240502 08:17:50.476639 13524 consensus_queue.cc:260] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 385, Last appended: 1.385, Last appended by leader: 385, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 385 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.477219 13524 ts_tablet_manager.cc:1398] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:17:50.478610 10519 tablet_copy_service.cc:342] P 750dbb32b0114697b81aeb7fca556e7f: Request end of tablet copy session ca969bc50f72467b94d8436d37be2f8e-c0a4038683ac48449ccad5eaeeef5694 received from {username='slave'} at 127.6.151.72:57809
I20240502 08:17:50.478788 10519 tablet_copy_service.cc:433] P 750dbb32b0114697b81aeb7fca556e7f: ending tablet copy session ca969bc50f72467b94d8436d37be2f8e-c0a4038683ac48449ccad5eaeeef5694 on tablet c0a4038683ac48449ccad5eaeeef5694 with peer ca969bc50f72467b94d8436d37be2f8e
I20240502 08:17:50.684998 13550 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=398 overwritten=0 applied=398 ignored=0} inserts{seen=1026 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:50.685442 13550 tablet_bootstrap.cc:492] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap complete.
I20240502 08:17:50.685830 13550 ts_tablet_manager.cc:1367] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: Time spent bootstrapping tablet: real 0.599s	user 0.436s	sys 0.087s
I20240502 08:17:50.686740 13550 raft_consensus.cc:348] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.687148 13550 raft_consensus.cc:729] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca969bc50f72467b94d8436d37be2f8e, State: Initialized, Role: LEARNER
I20240502 08:17:50.687386 13550 consensus_queue.cc:260] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 398, Last appended: 1.398, Last appended by leader: 398, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 398 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.687985 13550 ts_tablet_manager.cc:1398] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:17:50.689046 10389 tablet_copy_service.cc:342] P f577199118d5411b98b759d6c430daae: Request end of tablet copy session ca969bc50f72467b94d8436d37be2f8e-2b27b7cc14814dc48c1fe37dbbea9619 received from {username='slave'} at 127.6.151.72:37271
I20240502 08:17:50.689189 10389 tablet_copy_service.cc:433] P f577199118d5411b98b759d6c430daae: ending tablet copy session ca969bc50f72467b94d8436d37be2f8e-2b27b7cc14814dc48c1fe37dbbea9619 on tablet 2b27b7cc14814dc48c1fe37dbbea9619 with peer ca969bc50f72467b94d8436d37be2f8e
I20240502 08:17:50.743139 13558 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=416 overwritten=0 applied=416 ignored=0} inserts{seen=1098 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:17:50.743563 13558 tablet_bootstrap.cc:492] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: Bootstrap complete.
I20240502 08:17:50.743846 13558 ts_tablet_manager.cc:1367] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: Time spent bootstrapping tablet: real 0.552s	user 0.437s	sys 0.048s
I20240502 08:17:50.744475 13558 raft_consensus.cc:348] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 416 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.744762 13558 raft_consensus.cc:729] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca969bc50f72467b94d8436d37be2f8e, State: Initialized, Role: LEARNER
I20240502 08:17:50.745012 13558 consensus_queue.cc:260] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 416, Last appended: 1.416, Last appended by leader: 416, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 416 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: true } }
I20240502 08:17:50.745718 13558 ts_tablet_manager.cc:1398] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:17:50.746608 10258 tablet_copy_service.cc:342] P 9279bdb860044c90ace75c2f03950e02: Request end of tablet copy session ca969bc50f72467b94d8436d37be2f8e-00d1cef3b67e4d58aa76c8fd23f2c099 received from {username='slave'} at 127.6.151.72:58491
I20240502 08:17:50.746783 10258 tablet_copy_service.cc:433] P 9279bdb860044c90ace75c2f03950e02: ending tablet copy session ca969bc50f72467b94d8436d37be2f8e-00d1cef3b67e4d58aa76c8fd23f2c099 on tablet 00d1cef3b67e4d58aa76c8fd23f2c099 with peer ca969bc50f72467b94d8436d37be2f8e
I20240502 08:17:50.893464 13308 raft_consensus.cc:1203] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.384->[1.385-1.385]   Dedup: 1.385->[]
I20240502 08:17:50.935376 13308 raft_consensus.cc:1203] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.320->[2.321-2.321]   Dedup: 2.321->[]
I20240502 08:17:50.996712 13308 raft_consensus.cc:1203] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.415->[1.416-1.416]   Dedup: 1.416->[]
I20240502 08:17:51.019129 13308 raft_consensus.cc:1203] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.397->[1.398-1.398]   Dedup: 1.398->[]
I20240502 08:17:51.276316 13576 raft_consensus.cc:1050] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: attempting to promote NON_VOTER ca969bc50f72467b94d8436d37be2f8e to VOTER
I20240502 08:17:51.277088 13576 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.279567 11799 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240502 08:17:51.279764 13308 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 1 index: 386. (index mismatch)
I20240502 08:17:51.280244 13576 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240502 08:17:51.280575 13396 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s
I20240502 08:17:51.283352 13396 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Committing config change with OpId 1.386: config changed from index 385 to 386, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.286688 11799 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.287832 10093 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 385 to 386, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:51.290400 13308 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 1 FOLLOWER]: Committing config change with OpId 1.386: config changed from index 385 to 386, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.325767 13396 raft_consensus.cc:1050] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f: attempting to promote NON_VOTER ca969bc50f72467b94d8436d37be2f8e to VOTER
I20240502 08:17:51.326395 13396 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 2.321, Last appended by leader: 218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.328395 11667 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch)
I20240502 08:17:51.328615 13308 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [term 2 LEARNER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch)
I20240502 08:17:51.328936 13574 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20240502 08:17:51.329339 13576 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s
I20240502 08:17:51.332108 13576 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 2 LEADER]: Committing config change with OpId 2.322: config changed from index 321 to 322, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.335147 13308 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 321 to 322, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.336897 10093 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 321 to 322, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:51.339372 11667 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 321 to 322, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false replace: true } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.342123 10499 consensus_queue.cc:237] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 2.322, Last appended by leader: 218, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.343827 13308 raft_consensus.cc:1261] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Refusing update from remote peer 750dbb32b0114697b81aeb7fca556e7f: Log matching property violated. Preceding OpId in replica: term: 2 index: 322. Preceding OpId from leader: term: 2 index: 323. (index mismatch)
I20240502 08:17:51.344332 13576 consensus_queue.cc:1035] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s
I20240502 08:17:51.346604 13396 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f [term 2 LEADER]: Committing config change with OpId 2.323: config changed from index 322 to 323, VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.347262 13308 raft_consensus.cc:2932] T b02fadb91da44d33bd8f45847433a1ab P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Committing config change with OpId 2.323: config changed from index 322 to 323, VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) evicted. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.349422 10083 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b02fadb91da44d33bd8f45847433a1ab with cas_config_opid_index 322: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:51.350775 10097 catalog_manager.cc:5484] T b02fadb91da44d33bd8f45847433a1ab P 750dbb32b0114697b81aeb7fca556e7f reported cstate change: config changed from index 322 to 323, VOTER 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69) evicted. New cstate: current_term: 2 leader_uuid: "750dbb32b0114697b81aeb7fca556e7f" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:51.367208 11650 tablet_service.cc:1514] Processing DeleteTablet for tablet b02fadb91da44d33bd8f45847433a1ab with delete_type TABLET_DATA_TOMBSTONED (TS 54f96b9fb77540e898821b386cf6d0cb not found in new config with opid_index 323) from {username='slave'} at 127.0.0.1:48948
I20240502 08:17:51.368093 13677 tablet_replica.cc:330] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: stopping tablet replica
I20240502 08:17:51.368458 13677 raft_consensus.cc:2229] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:51.368782 13677 raft_consensus.cc:2258] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:51.370800 13677 ts_tablet_manager.cc:1875] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:51.375752 13677 ts_tablet_manager.cc:1888] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.322
I20240502 08:17:51.375869 13677 log.cc:1196] T b02fadb91da44d33bd8f45847433a1ab P 54f96b9fb77540e898821b386cf6d0cb: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/wals/b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:51.376703 10082 catalog_manager.cc:4830] TS 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733): tablet b02fadb91da44d33bd8f45847433a1ab (table rebalance_test_table_2 [id=3a203aa4e6454c2d86e244e14f43fed7]) successfully deleted
I20240502 08:17:51.448617 13466 raft_consensus.cc:1050] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: attempting to promote NON_VOTER ca969bc50f72467b94d8436d37be2f8e to VOTER
I20240502 08:17:51.449442 13466 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 416, Committed index: 416, Last appended: 1.416, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.451696 11533 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 416. Preceding OpId from leader: term: 1 index: 417. (index mismatch)
I20240502 08:17:51.451695 13308 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Refusing update from remote peer 9279bdb860044c90ace75c2f03950e02: Log matching property violated. Preceding OpId in replica: term: 1 index: 416. Preceding OpId from leader: term: 1 index: 417. (index mismatch)
I20240502 08:17:51.452260 13622 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 417, Last known committed idx: 416, Time since last communication: 0.000s
I20240502 08:17:51.452592 13587 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 417, Last known committed idx: 416, Time since last communication: 0.000s
I20240502 08:17:51.455251 13466 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Committing config change with OpId 1.417: config changed from index 416 to 417, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.456176 13308 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 1 FOLLOWER]: Committing config change with OpId 1.417: config changed from index 416 to 417, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.458472 11533 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Committing config change with OpId 1.417: config changed from index 416 to 417, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.459888 10097 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 reported cstate change: config changed from index 416 to 417, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9279bdb860044c90ace75c2f03950e02" committed_config { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:51.512905 13473 raft_consensus.cc:1050] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: attempting to promote NON_VOTER ca969bc50f72467b94d8436d37be2f8e to VOTER
I20240502 08:17:51.513818 13473 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 398, Committed index: 398, Last appended: 1.398, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.516597 13308 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 1 LEARNER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 398. Preceding OpId from leader: term: 1 index: 399. (index mismatch)
I20240502 08:17:51.516829 11667 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Refusing update from remote peer f577199118d5411b98b759d6c430daae: Log matching property violated. Preceding OpId in replica: term: 1 index: 398. Preceding OpId from leader: term: 1 index: 399. (index mismatch)
I20240502 08:17:51.517131 13473 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I20240502 08:17:51.517470 13601 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [LEADER]: Connected to new peer: Peer: permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 399, Last known committed idx: 398, Time since last communication: 0.000s
I20240502 08:17:51.517748 10499 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0a4038683ac48449ccad5eaeeef5694"
dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60732
I20240502 08:17:51.517963 10499 raft_consensus.cc:595] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:51.521034 13601 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Committing config change with OpId 1.399: config changed from index 398 to 399, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.521757 13308 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 1 FOLLOWER]: Committing config change with OpId 1.399: config changed from index 398 to 399, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.522567 11667 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Committing config change with OpId 1.399: config changed from index 398 to 399, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:51.527150 10096 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae reported cstate change: config changed from index 398 to 399, ca969bc50f72467b94d8436d37be2f8e (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f577199118d5411b98b759d6c430daae" committed_config { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:51.548085 10235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00d1cef3b67e4d58aa76c8fd23f2c099"
dest_uuid: "9279bdb860044c90ace75c2f03950e02"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33484
I20240502 08:17:51.548359 10235 raft_consensus.cc:595] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:51.566146 10368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b27b7cc14814dc48c1fe37dbbea9619"
dest_uuid: "f577199118d5411b98b759d6c430daae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37686
I20240502 08:17:51.566437 10368 raft_consensus.cc:595] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:51.584017 10499 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0a4038683ac48449ccad5eaeeef5694"
dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:60748
I20240502 08:17:51.584234 10499 raft_consensus.cc:595] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:51.600792 10235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00d1cef3b67e4d58aa76c8fd23f2c099"
dest_uuid: "9279bdb860044c90ace75c2f03950e02"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:33514
I20240502 08:17:51.601024 10235 raft_consensus.cc:595] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:51.799489 13396 raft_consensus.cc:979] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: : Instructing follower ca969bc50f72467b94d8436d37be2f8e to start an election
I20240502 08:17:51.799743 13576 raft_consensus.cc:1067] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Signalling peer ca969bc50f72467b94d8436d37be2f8e to start an election
I20240502 08:17:51.800637 13308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0a4038683ac48449ccad5eaeeef5694"
dest_uuid: "ca969bc50f72467b94d8436d37be2f8e"
 from {username='slave'} at 127.6.151.67:46877
I20240502 08:17:51.800904 13308 raft_consensus.cc:482] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:51.801065 13308 raft_consensus.cc:3037] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.803193 13308 raft_consensus.cc:504] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.805299 10499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a4038683ac48449ccad5eaeeef5694" candidate_uuid: "ca969bc50f72467b94d8436d37be2f8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 386 } } ignore_live_leader: true dest_uuid: "750dbb32b0114697b81aeb7fca556e7f"
I20240502 08:17:51.805502 10499 raft_consensus.cc:3032] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:51.805588 10499 raft_consensus.cc:729] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 750dbb32b0114697b81aeb7fca556e7f, State: Running, Role: LEADER
I20240502 08:17:51.805852 10499 consensus_queue.cc:260] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 1.386, Last appended by leader: 386, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.806253 10499 raft_consensus.cc:3037] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.808295 10499 raft_consensus.cc:2454] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca969bc50f72467b94d8436d37be2f8e in term 2.
I20240502 08:17:51.808583 13308 leader_election.cc:288] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [CANDIDATE]: Term 2 election: Requested vote from peers 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465), a57c9c479b194a69b5eb8f9361819a83 (127.6.151.70:33091)
I20240502 08:17:51.808933 13243 leader_election.cc:302] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 750dbb32b0114697b81aeb7fca556e7f, ca969bc50f72467b94d8436d37be2f8e; no voters: 
I20240502 08:17:51.809401 13580 raft_consensus.cc:2781] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:51.811105 13580 raft_consensus.cc:686] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 2 LEADER]: Becoming Leader. State: Replica: ca969bc50f72467b94d8436d37be2f8e, State: Running, Role: LEADER
I20240502 08:17:51.811720 13580 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 1.386, Last appended by leader: 386, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.816448 10093 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e reported cstate change: term changed from 1 to 2, leader changed from 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) to ca969bc50f72467b94d8436d37be2f8e (127.6.151.72). New cstate: current_term: 2 leader_uuid: "ca969bc50f72467b94d8436d37be2f8e" committed_config { opid_index: 386 OBSOLETE_local: false peers { permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:51.817387 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0a4038683ac48449ccad5eaeeef5694" candidate_uuid: "ca969bc50f72467b94d8436d37be2f8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 386 } } ignore_live_leader: true dest_uuid: "a57c9c479b194a69b5eb8f9361819a83"
I20240502 08:17:51.817615 11799 raft_consensus.cc:3037] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.820021 11799 raft_consensus.cc:2454] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca969bc50f72467b94d8436d37be2f8e in term 2.
I20240502 08:17:51.833206 13587 raft_consensus.cc:979] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: : Instructing follower 2fac43dcafe6465b941acc57014c43d3 to start an election
I20240502 08:17:51.833432 13587 raft_consensus.cc:1067] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Signalling peer 2fac43dcafe6465b941acc57014c43d3 to start an election
I20240502 08:17:51.834156 10368 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2b27b7cc14814dc48c1fe37dbbea9619"
dest_uuid: "f577199118d5411b98b759d6c430daae"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37706
I20240502 08:17:51.834362 10368 raft_consensus.cc:595] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Received request to transfer leadership
I20240502 08:17:51.834391 11533 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00d1cef3b67e4d58aa76c8fd23f2c099"
dest_uuid: "2fac43dcafe6465b941acc57014c43d3"
 from {username='slave'} at 127.6.151.65:47099
I20240502 08:17:51.834694 11533 raft_consensus.cc:482] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:51.834836 11533 raft_consensus.cc:3037] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.836954 11533 raft_consensus.cc:504] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.837877 11533 leader_election.cc:288] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 2 election: Requested vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271), ca969bc50f72467b94d8436d37be2f8e (127.6.151.72:34005)
I20240502 08:17:51.838294 10235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d1cef3b67e4d58aa76c8fd23f2c099" candidate_uuid: "2fac43dcafe6465b941acc57014c43d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 417 } } ignore_live_leader: true dest_uuid: "9279bdb860044c90ace75c2f03950e02"
I20240502 08:17:51.838536 10235 raft_consensus.cc:3032] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:51.838632 10235 raft_consensus.cc:729] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9279bdb860044c90ace75c2f03950e02, State: Running, Role: LEADER
I20240502 08:17:51.839025 10235 consensus_queue.cc:260] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 417, Committed index: 417, Last appended: 1.417, Last appended by leader: 417, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.839334 10235 raft_consensus.cc:3037] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.841420 10235 raft_consensus.cc:2454] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fac43dcafe6465b941acc57014c43d3 in term 2.
I20240502 08:17:51.841874 11468 leader_election.cc:302] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fac43dcafe6465b941acc57014c43d3, 9279bdb860044c90ace75c2f03950e02; no voters: 
I20240502 08:17:51.842186 13567 raft_consensus.cc:2781] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:51.844419 13567 raft_consensus.cc:686] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 2 LEADER]: Becoming Leader. State: Replica: 2fac43dcafe6465b941acc57014c43d3, State: Running, Role: LEADER
I20240502 08:17:51.844818 13567 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 417, Committed index: 417, Last appended: 1.417, Last appended by leader: 417, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.847738 10093 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 reported cstate change: term changed from 1 to 2, leader changed from 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) to 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68). New cstate: current_term: 2 leader_uuid: "2fac43dcafe6465b941acc57014c43d3" committed_config { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:51.852169 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00d1cef3b67e4d58aa76c8fd23f2c099" candidate_uuid: "2fac43dcafe6465b941acc57014c43d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 417 } } ignore_live_leader: true dest_uuid: "ca969bc50f72467b94d8436d37be2f8e"
I20240502 08:17:51.852504 13308 raft_consensus.cc:3037] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.855573 13308 raft_consensus.cc:2454] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fac43dcafe6465b941acc57014c43d3 in term 2.
I20240502 08:17:51.911010 13601 raft_consensus.cc:979] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: : Instructing follower 54f96b9fb77540e898821b386cf6d0cb to start an election
I20240502 08:17:51.911233 13601 raft_consensus.cc:1067] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Signalling peer 54f96b9fb77540e898821b386cf6d0cb to start an election
I20240502 08:17:51.911962 11667 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b27b7cc14814dc48c1fe37dbbea9619"
dest_uuid: "54f96b9fb77540e898821b386cf6d0cb"
 from {username='slave'} at 127.6.151.66:35379
I20240502 08:17:51.912256 11667 raft_consensus.cc:482] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:17:51.912410 11667 raft_consensus.cc:3037] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.915455 11667 raft_consensus.cc:504] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.917634 10368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b27b7cc14814dc48c1fe37dbbea9619" candidate_uuid: "54f96b9fb77540e898821b386cf6d0cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 399 } } ignore_live_leader: true dest_uuid: "f577199118d5411b98b759d6c430daae"
I20240502 08:17:51.917876 10368 raft_consensus.cc:3032] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:17:51.917882 11667 leader_election.cc:288] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [CANDIDATE]: Term 2 election: Requested vote from peers f577199118d5411b98b759d6c430daae (127.6.151.66:41685), ca969bc50f72467b94d8436d37be2f8e (127.6.151.72:34005)
I20240502 08:17:51.917958 10368 raft_consensus.cc:729] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f577199118d5411b98b759d6c430daae, State: Running, Role: LEADER
I20240502 08:17:51.918272 10368 consensus_queue.cc:260] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 1.399, Last appended by leader: 399, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.918633 10368 raft_consensus.cc:3037] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.921427 10368 raft_consensus.cc:2454] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f96b9fb77540e898821b386cf6d0cb in term 2.
I20240502 08:17:51.922041 11602 leader_election.cc:302] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54f96b9fb77540e898821b386cf6d0cb, f577199118d5411b98b759d6c430daae; no voters: 
I20240502 08:17:51.922435 13674 raft_consensus.cc:2781] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:17:51.926347 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b27b7cc14814dc48c1fe37dbbea9619" candidate_uuid: "54f96b9fb77540e898821b386cf6d0cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 399 } } ignore_live_leader: true dest_uuid: "ca969bc50f72467b94d8436d37be2f8e"
I20240502 08:17:51.926610 13308 raft_consensus.cc:3037] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:17:51.930601 13308 raft_consensus.cc:2454] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54f96b9fb77540e898821b386cf6d0cb in term 2.
I20240502 08:17:51.933625 13674 raft_consensus.cc:686] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 2 LEADER]: Becoming Leader. State: Replica: 54f96b9fb77540e898821b386cf6d0cb, State: Running, Role: LEADER
I20240502 08:17:51.934099 13674 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 399, Committed index: 399, Last appended: 1.399, Last appended by leader: 399, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:51.937386 10093 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb reported cstate change: term changed from 1 to 2, leader changed from f577199118d5411b98b759d6c430daae (127.6.151.66) to 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69). New cstate: current_term: 2 leader_uuid: "54f96b9fb77540e898821b386cf6d0cb" committed_config { opid_index: 399 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:17:52.228879 10235 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Refusing update from remote peer 2fac43dcafe6465b941acc57014c43d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 417. Preceding OpId from leader: term: 2 index: 418. (index mismatch)
I20240502 08:17:52.229543 13567 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 418, Last known committed idx: 417, Time since last communication: 0.000s
I20240502 08:17:52.235060 13308 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Refusing update from remote peer 2fac43dcafe6465b941acc57014c43d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 417. Preceding OpId from leader: term: 2 index: 418. (index mismatch)
I20240502 08:17:52.235821 13759 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 418, Last known committed idx: 417, Time since last communication: 0.000s
I20240502 08:17:52.312273 10499 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Refusing update from remote peer ca969bc50f72467b94d8436d37be2f8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 2 index: 387. (index mismatch)
I20240502 08:17:52.313344 13580 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "750dbb32b0114697b81aeb7fca556e7f" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 38465 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240502 08:17:52.319718 11799 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 2 FOLLOWER]: Refusing update from remote peer ca969bc50f72467b94d8436d37be2f8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 2 index: 387. (index mismatch)
I20240502 08:17:52.320955 13806 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.001s
I20240502 08:17:52.415385 10368 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Refusing update from remote peer 54f96b9fb77540e898821b386cf6d0cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 2 index: 400. (index mismatch)
I20240502 08:17:52.416185 13674 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20240502 08:17:52.420419 13308 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Refusing update from remote peer 54f96b9fb77540e898821b386cf6d0cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 399. Preceding OpId from leader: term: 2 index: 400. (index mismatch)
I20240502 08:17:52.421026 13674 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 400, Last known committed idx: 399, Time since last communication: 0.000s
I20240502 08:17:52.681530 11533 consensus_queue.cc:237] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 418, Committed index: 418, Last appended: 2.418, Last appended by leader: 417, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 419 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:52.683276 13308 raft_consensus.cc:1261] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Refusing update from remote peer 2fac43dcafe6465b941acc57014c43d3: Log matching property violated. Preceding OpId in replica: term: 2 index: 418. Preceding OpId from leader: term: 2 index: 419. (index mismatch)
I20240502 08:17:52.683843 13832 consensus_queue.cc:1035] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 419, Last known committed idx: 418, Time since last communication: 0.000s
I20240502 08:17:52.686215 13567 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 [term 2 LEADER]: Committing config change with OpId 2.419: config changed from index 417 to 419, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 419 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:52.686883 13308 raft_consensus.cc:2932] T 00d1cef3b67e4d58aa76c8fd23f2c099 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Committing config change with OpId 2.419: config changed from index 417 to 419, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New config: { opid_index: 419 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:52.690315 10081 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 with cas_config_opid_index 417: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:52.692086 10096 catalog_manager.cc:5484] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 2fac43dcafe6465b941acc57014c43d3 reported cstate change: config changed from index 417 to 419, VOTER 9279bdb860044c90ace75c2f03950e02 (127.6.151.65) evicted. New cstate: current_term: 2 leader_uuid: "2fac43dcafe6465b941acc57014c43d3" committed_config { opid_index: 419 OBSOLETE_local: false peers { permanent_uuid: "2fac43dcafe6465b941acc57014c43d3" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 37265 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:52.696693 10217 tablet_service.cc:1514] Processing DeleteTablet for tablet 00d1cef3b67e4d58aa76c8fd23f2c099 with delete_type TABLET_DATA_TOMBSTONED (TS 9279bdb860044c90ace75c2f03950e02 not found in new config with opid_index 419) from {username='slave'} at 127.0.0.1:34168
I20240502 08:17:52.703711 13837 tablet_replica.cc:330] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: stopping tablet replica
I20240502 08:17:52.703938 13837 raft_consensus.cc:2229] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.704139 13837 raft_consensus.cc:2258] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.706240 13837 ts_tablet_manager.cc:1875] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:52.713099 13837 ts_tablet_manager.cc:1888] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.418
I20240502 08:17:52.713292 13837 log.cc:1196] T 00d1cef3b67e4d58aa76c8fd23f2c099 P 9279bdb860044c90ace75c2f03950e02: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:52.714321 10082 catalog_manager.cc:4830] TS 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): tablet 00d1cef3b67e4d58aa76c8fd23f2c099 (table rebalance_test_table_3 [id=ee2d386900364032847f6f2be742b075]) successfully deleted
I20240502 08:17:52.906351 13308 consensus_queue.cc:237] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 2.387, Last appended by leader: 386, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:52.908216 11799 raft_consensus.cc:1261] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 2 FOLLOWER]: Refusing update from remote peer ca969bc50f72467b94d8436d37be2f8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 387. Preceding OpId from leader: term: 2 index: 388. (index mismatch)
I20240502 08:17:52.908989 13863 consensus_queue.cc:1035] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240502 08:17:52.911648 13805 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e [term 2 LEADER]: Committing config change with OpId 2.388: config changed from index 386 to 388, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:52.912362 11799 raft_consensus.cc:2932] T c0a4038683ac48449ccad5eaeeef5694 P a57c9c479b194a69b5eb8f9361819a83 [term 2 FOLLOWER]: Committing config change with OpId 2.388: config changed from index 386 to 388, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:52.915895 10084 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0a4038683ac48449ccad5eaeeef5694 with cas_config_opid_index 386: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:52.918061 10096 catalog_manager.cc:5484] T c0a4038683ac48449ccad5eaeeef5694 P ca969bc50f72467b94d8436d37be2f8e reported cstate change: config changed from index 386 to 388, VOTER 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67) evicted. New cstate: current_term: 2 leader_uuid: "ca969bc50f72467b94d8436d37be2f8e" committed_config { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:52.922346 10479 tablet_service.cc:1514] Processing DeleteTablet for tablet c0a4038683ac48449ccad5eaeeef5694 with delete_type TABLET_DATA_TOMBSTONED (TS 750dbb32b0114697b81aeb7fca556e7f not found in new config with opid_index 388) from {username='slave'} at 127.0.0.1:46754
I20240502 08:17:52.925799 13864 tablet_replica.cc:330] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: stopping tablet replica
I20240502 08:17:52.926055 13864 raft_consensus.cc:2229] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:52.926277 13864 raft_consensus.cc:2258] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:52.928467 13864 ts_tablet_manager.cc:1875] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:52.935488 13864 ts_tablet_manager.cc:1888] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.387
I20240502 08:17:52.935640 13864 log.cc:1196] T c0a4038683ac48449ccad5eaeeef5694 P 750dbb32b0114697b81aeb7fca556e7f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:52.936558 10083 catalog_manager.cc:4830] TS 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465): tablet c0a4038683ac48449ccad5eaeeef5694 (table rebalance_test_table_4 [id=acb50ae09a204676894c5546e915747c]) successfully deleted
I20240502 08:17:53.031107 11667 consensus_queue.cc:237] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 2.400, Last appended by leader: 399, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } }
I20240502 08:17:53.033457 13308 raft_consensus.cc:1261] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Refusing update from remote peer 54f96b9fb77540e898821b386cf6d0cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 400. Preceding OpId from leader: term: 2 index: 401. (index mismatch)
I20240502 08:17:53.034050 13867 consensus_queue.cc:1035] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 401, Last known committed idx: 400, Time since last communication: 0.000s
I20240502 08:17:53.037328 13866 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb [term 2 LEADER]: Committing config change with OpId 2.401: config changed from index 399 to 401, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:53.038095 13308 raft_consensus.cc:2932] T 2b27b7cc14814dc48c1fe37dbbea9619 P ca969bc50f72467b94d8436d37be2f8e [term 2 FOLLOWER]: Committing config change with OpId 2.401: config changed from index 399 to 401, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New config: { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } } }
I20240502 08:17:53.041909 10082 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2b27b7cc14814dc48c1fe37dbbea9619 with cas_config_opid_index 399: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:17:53.043517 10096 catalog_manager.cc:5484] T 2b27b7cc14814dc48c1fe37dbbea9619 P 54f96b9fb77540e898821b386cf6d0cb reported cstate change: config changed from index 399 to 401, VOTER f577199118d5411b98b759d6c430daae (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "54f96b9fb77540e898821b386cf6d0cb" committed_config { opid_index: 401 OBSOLETE_local: false peers { permanent_uuid: "54f96b9fb77540e898821b386cf6d0cb" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 44733 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca969bc50f72467b94d8436d37be2f8e" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 34005 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:17:53.047602 10348 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b27b7cc14814dc48c1fe37dbbea9619 with delete_type TABLET_DATA_TOMBSTONED (TS f577199118d5411b98b759d6c430daae not found in new config with opid_index 401) from {username='slave'} at 127.0.0.1:52194
I20240502 08:17:53.048383 13876 tablet_replica.cc:330] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: stopping tablet replica
I20240502 08:17:53.048616 13876 raft_consensus.cc:2229] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:17:53.048877 13876 raft_consensus.cc:2258] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:17:53.051492 13876 ts_tablet_manager.cc:1875] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:17:53.057885 13876 ts_tablet_manager.cc:1888] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.400
I20240502 08:17:53.058059 13876 log.cc:1196] T 2b27b7cc14814dc48c1fe37dbbea9619 P f577199118d5411b98b759d6c430daae: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:53.059039 10082 catalog_manager.cc:4830] TS f577199118d5411b98b759d6c430daae (127.6.151.66:41685): tablet 2b27b7cc14814dc48c1fe37dbbea9619 (table rebalance_test_table_1 [id=ce557d42789346aba2fe05b973099e77]) successfully deleted
I20240502 08:17:53.172240 11515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.192162 10479 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:53.194243 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.221787 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.256453 11650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.260241 13288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.282765 10217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.296583 10348 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:53.469417 11515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.485206 10479 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:53.486225 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.490703 11650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.525462 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.528932 10217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.540205 13288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.545266 10348 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:17:53.861769 11515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.876502 11650 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.895120 10217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.907686 10479 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:17:53.919209 11780 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.944204 11919 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:17:53.971632 13288 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:17:53.995625 10348 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 108c4a65ead24022acd2eaae8cc9aece | 127.6.151.126:36085 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                Value                                                                                |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 allow_unsafe_replication_factor          | true                                                                                                                                                                | runtime,unsafe        | all 1 server(s) checked
 ipki_ca_key_size                         | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                 | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                   | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                  | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                 | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:35619 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 2fac43dcafe6465b941acc57014c43d3 | 127.6.151.68:37265 | HEALTHY | <none>   |       2        |       0
 54f96b9fb77540e898821b386cf6d0cb | 127.6.151.69:44733 | HEALTHY | <none>   |       1        |       0
 750dbb32b0114697b81aeb7fca556e7f | 127.6.151.67:38465 | HEALTHY | <none>   |       3        |       0
 84379f2079ff403eacad196d57abd7ad | 127.6.151.71:43045 | HEALTHY | <none>   |       2        |       0
 9279bdb860044c90ace75c2f03950e02 | 127.6.151.65:43271 | HEALTHY | <none>   |       2        |       0
 a57c9c479b194a69b5eb8f9361819a83 | 127.6.151.70:33091 | HEALTHY | <none>   |       1        |       0
 ca969bc50f72467b94d8436d37be2f8e | 127.6.151.72:34005 | HEALTHY | <none>   |       1        |       0
 f577199118d5411b98b759d6c430daae | 127.6.151.66:41685 | HEALTHY | <none>   |       4        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       8

Unusual flags for Tablet Server:
                   Flag                   |                                                                              Value                                                                              |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                             | experimental          | all 8 server(s) checked
 local_ip_for_outbound_sockets            | 127.6.151.65                                                                                                                                                    | experimental          | 127.6.151.65:43271
 local_ip_for_outbound_sockets            | 127.6.151.66                                                                                                                                                    | experimental          | 127.6.151.66:41685
 local_ip_for_outbound_sockets            | 127.6.151.67                                                                                                                                                    | experimental          | 127.6.151.67:38465
 local_ip_for_outbound_sockets            | 127.6.151.68                                                                                                                                                    | experimental          | 127.6.151.68:37265
 local_ip_for_outbound_sockets            | 127.6.151.69                                                                                                                                                    | experimental          | 127.6.151.69:44733
 local_ip_for_outbound_sockets            | 127.6.151.70                                                                                                                                                    | experimental          | 127.6.151.70:33091
 local_ip_for_outbound_sockets            | 127.6.151.71                                                                                                                                                    | experimental          | 127.6.151.71:43045
 local_ip_for_outbound_sockets            | 127.6.151.72                                                                                                                                                    | experimental          | 127.6.151.72:34005
 never_fsync                              | true                                                                                                                                                            | unsafe,advanced       | all 8 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                               | unsafe,hidden         | all 8 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                            | experimental,advanced | all 8 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                            | unsafe                | all 8 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                              | hidden                | all 8 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.6.151.65:43271
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.6.151.66:41685
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.6.151.67:38465
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.6.151.68:37265
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden                | 127.6.151.69:44733
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden                | 127.6.151.70:33091
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden                | 127.6.151.71:43045
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden                | 127.6.151.72:34005

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:35619 | all 8 server(s) checked
 time_source         | builtin            | all 8 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 9 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 2  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_4 | 2  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 4
 First Quartile | 4
 Median         | 4
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 8
 Tables         | 6
 Tablets        | 16
 Replicas       | 32

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 08:17:54.055650  6749 log_verifier.cc:126] Checking tablet 00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:54.215132  6749 log_verifier.cc:177] Verified matching terms for 419 ops in tablet 00d1cef3b67e4d58aa76c8fd23f2c099
I20240502 08:17:54.215692  6749 log_verifier.cc:126] Checking tablet 138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:54.343595  6749 log_verifier.cc:177] Verified matching terms for 358 ops in tablet 138b1526ab364e6bafe0abfa1edb4376
I20240502 08:17:54.344062  6749 log_verifier.cc:126] Checking tablet 21631366c38a4a608cb8191903fc535d
I20240502 08:17:54.488214  6749 log_verifier.cc:177] Verified matching terms for 350 ops in tablet 21631366c38a4a608cb8191903fc535d
I20240502 08:17:54.488956  6749 log_verifier.cc:126] Checking tablet 2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:54.631772  6749 log_verifier.cc:177] Verified matching terms for 401 ops in tablet 2b27b7cc14814dc48c1fe37dbbea9619
I20240502 08:17:54.632251  6749 log_verifier.cc:126] Checking tablet 6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:54.770787  6749 log_verifier.cc:177] Verified matching terms for 324 ops in tablet 6fd1419d08a94610b725cc1ed2b04d00
I20240502 08:17:54.771299  6749 log_verifier.cc:126] Checking tablet 75f37a6f3a254fec8b7805185d9ae80f
I20240502 08:17:54.912056  6749 log_verifier.cc:177] Verified matching terms for 341 ops in tablet 75f37a6f3a254fec8b7805185d9ae80f
I20240502 08:17:54.912546  6749 log_verifier.cc:126] Checking tablet 82835c215cce4ce49680d56a5bbf0208
I20240502 08:17:55.063866  6749 log_verifier.cc:177] Verified matching terms for 401 ops in tablet 82835c215cce4ce49680d56a5bbf0208
I20240502 08:17:55.064365  6749 log_verifier.cc:126] Checking tablet a2c5cbee8e5546739f31b047565e1144
I20240502 08:17:55.192648  6749 log_verifier.cc:177] Verified matching terms for 362 ops in tablet a2c5cbee8e5546739f31b047565e1144
I20240502 08:17:55.193156  6749 log_verifier.cc:126] Checking tablet b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:55.318059  6749 log_verifier.cc:177] Verified matching terms for 323 ops in tablet b02fadb91da44d33bd8f45847433a1ab
I20240502 08:17:55.318485  6749 log_verifier.cc:126] Checking tablet bde4a6ed1250456696a769d280e19ac4
I20240502 08:17:55.481907  6749 log_verifier.cc:177] Verified matching terms for 373 ops in tablet bde4a6ed1250456696a769d280e19ac4
I20240502 08:17:55.482721  6749 log_verifier.cc:126] Checking tablet befc5d75417d47d8b7790017c49fcb71
I20240502 08:17:55.634289  6749 log_verifier.cc:177] Verified matching terms for 431 ops in tablet befc5d75417d47d8b7790017c49fcb71
I20240502 08:17:55.634809  6749 log_verifier.cc:126] Checking tablet c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:55.776408  6749 log_verifier.cc:177] Verified matching terms for 388 ops in tablet c0a4038683ac48449ccad5eaeeef5694
I20240502 08:17:55.777146  6749 log_verifier.cc:126] Checking tablet d426a029b71e4dd4a8852a5cfe708ae0
I20240502 08:17:55.912781  6749 log_verifier.cc:177] Verified matching terms for 348 ops in tablet d426a029b71e4dd4a8852a5cfe708ae0
I20240502 08:17:55.913508  6749 log_verifier.cc:126] Checking tablet d471c5fb3045495aa1ec0a774e6164d8
I20240502 08:17:56.050454  6749 log_verifier.cc:177] Verified matching terms for 332 ops in tablet d471c5fb3045495aa1ec0a774e6164d8
I20240502 08:17:56.050959  6749 log_verifier.cc:126] Checking tablet f5f8034ce87a4c85a72c53ba07f178d1
I20240502 08:17:56.062270  6749 log_verifier.cc:177] Verified matching terms for 2 ops in tablet f5f8034ce87a4c85a72c53ba07f178d1
I20240502 08:17:56.062364  6749 log_verifier.cc:126] Checking tablet fb8df38d1c3340fda6024301f0798487
I20240502 08:17:56.197665  6749 log_verifier.cc:177] Verified matching terms for 380 ops in tablet fb8df38d1c3340fda6024301f0798487
I20240502 08:17:56.205060  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10156
I20240502 08:17:56.206282 10278 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:56.438071  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10156
W20240502 08:17:56.482312 10303 connection.cc:535] client connection to 127.6.151.65:43271 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:56.482371 10083 connection.cc:535] server connection from 127.6.151.65:37397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:56.482605 10303 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:56.482854  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10287
I20240502 08:17:56.483870 10409 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:56.729293 10303 consensus_peers.cc:485] T d471c5fb3045495aa1ec0a774e6164d8 P f577199118d5411b98b759d6c430daae -> Peer 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): Couldn't send request to peer 9279bdb860044c90ace75c2f03950e02. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:43271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:56.729351  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10287
W20240502 08:17:56.774583 11602 connection.cc:535] server connection from 127.6.151.66:35379 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:56.775141  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10417
I20240502 08:17:56.776126 10539 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:56.866604 11734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.65:43271: connect: Connection refused (error 111)
W20240502 08:17:56.870060 11734 consensus_peers.cc:485] T 21631366c38a4a608cb8191903fc535d P a57c9c479b194a69b5eb8f9361819a83 -> Peer 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): Couldn't send request to peer 9279bdb860044c90ace75c2f03950e02. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:43271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:57.001892  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10417
W20240502 08:17:57.041627 11469 connection.cc:535] client connection to 127.6.151.67:38465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:17:57.041963 11469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:57.042426  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 11452
W20240502 08:17:57.045099 11469 consensus_peers.cc:485] T f5f8034ce87a4c85a72c53ba07f178d1 P 2fac43dcafe6465b941acc57014c43d3 -> Peer 750dbb32b0114697b81aeb7fca556e7f (127.6.151.67:38465): Couldn't send request to peer 750dbb32b0114697b81aeb7fca556e7f. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.67:38465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:57.049517 11576 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.206415  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 11452
I20240502 08:17:57.236145  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 11586
I20240502 08:17:57.243006 11709 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.420053  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 11586
W20240502 08:17:57.451143 13241 connection.cc:535] server connection from 127.6.151.69:38533 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:17:57.451629  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 11718
W20240502 08:17:57.451611 11873 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.151.69:44733 (error 108)
W20240502 08:17:57.453614 11873 consensus_peers.cc:485] T 138b1526ab364e6bafe0abfa1edb4376 P 84379f2079ff403eacad196d57abd7ad -> Peer 54f96b9fb77540e898821b386cf6d0cb (127.6.151.69:44733): Couldn't send request to peer 54f96b9fb77540e898821b386cf6d0cb. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:44733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:57.457926 11839 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:17:57.538705 11872 consensus_peers.cc:485] T 6fd1419d08a94610b725cc1ed2b04d00 P 84379f2079ff403eacad196d57abd7ad -> Peer 2fac43dcafe6465b941acc57014c43d3 (127.6.151.68:37265): Couldn't send request to peer 2fac43dcafe6465b941acc57014c43d3. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.68:37265: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:17:57.561368 13573 raft_consensus.cc:482] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9279bdb860044c90ace75c2f03950e02)
I20240502 08:17:57.561591 13573 raft_consensus.cc:504] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "9279bdb860044c90ace75c2f03950e02" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43271 } } peers { permanent_uuid: "84379f2079ff403eacad196d57abd7ad" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 43045 } attrs { promote: false } }
I20240502 08:17:57.562345 13573 leader_election.cc:288] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271)
W20240502 08:17:57.563635 11873 leader_election.cc:334] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9279bdb860044c90ace75c2f03950e02 (127.6.151.65:43271): Network error: Client connection negotiation failed: client connection to 127.6.151.65:43271: connect: Connection refused (error 111)
I20240502 08:17:57.563761 11873 leader_election.cc:302] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 84379f2079ff403eacad196d57abd7ad; no voters: 9279bdb860044c90ace75c2f03950e02
I20240502 08:17:57.564033 13573 raft_consensus.cc:2726] T befc5d75417d47d8b7790017c49fcb71 P 84379f2079ff403eacad196d57abd7ad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 08:17:57.623351 14015 raft_consensus.cc:482] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f577199118d5411b98b759d6c430daae)
I20240502 08:17:57.623582 14015 raft_consensus.cc:504] T a2c5cbee8e5546739f31b047565e1144 P a57c9c479b194a69b5eb8f9361819a83 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "f577199118d5411b98b759d6c430daae" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 41685 } } peers { permanent_uuid: "a57c9c479b194a69b5eb8f9361819a83" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 33091 } attrs { promote: false } }
I20240502 08:17:57.623893  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 11718
I20240502 08:17:57.653841  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 11848
I20240502 08:17:57.660118 11979 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.827354  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 11848
I20240502 08:17:57.857743  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 13224
I20240502 08:17:57.863790 13347 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:57.992380  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 13224
I20240502 08:17:58.014864  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 10066
I20240502 08:17:58.015980 10126 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:17:58.180992  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 10066
2024-05-02T08:17:58Z chronyd exiting
[       OK ] RebalanceParamTest.Rebalance/3 (50497 ms)
[----------] 1 test from RebalanceParamTest (50497 ms total)

[----------] 1 test from ConcurrentRebalancersTest
[ RUN      ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240502 08:17:58.272039  6749 test_util.cc:268] Using random seed: 1034362903
I20240502 08:17:58.273744  6749 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:17:58.273797  6749 ts_itest-base.cc:113] --------------
I20240502 08:17:58.273818  6749 ts_itest-base.cc:114] 8 tablet servers
I20240502 08:17:58.273839  6749 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:17:58.273877  6749 ts_itest-base.cc:116] --------------
2024-05-02T08:17:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:17:58Z Disabled control of system clock
I20240502 08:17:58.327831  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:35417
--webserver_interface=127.6.151.126
--webserver_port=0
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.151.126:35417
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 08:17:58.675803 14034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:58.676112 14034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:58.676242 14034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:58.683820 14034 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:58.683948 14034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:17:58.684046 14034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:58.684123 14034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:17:58.684197 14034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:17:58.693346 14034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.151.126:35417
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.151.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:58.694633 14034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:58.696372 14034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:58.707291 14039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:58.707684 14040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:58.707924 14042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:58.708525 14034 server_base.cc:1034] running on GCE node
I20240502 08:17:58.709277 14034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:58.710633 14034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:58.711869 14034 hybrid_clock.cc:648] HybridClock initialized: now 1714637878711828 us; error 36 us; skew 500 ppm
I20240502 08:17:58.712200 14034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:58.718695 14034 webserver.cc:460] Webserver started at http://127.6.151.126:39549/ using document root <none> and password file <none>
I20240502 08:17:58.719466 14034 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:58.719571 14034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:58.719990 14034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:58.722713 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c6b8f0f2531f40e9932bec32d4bd4988"
format_stamp: "Formatted at 2024-05-02 08:17:58 on dist-test-slave-x3qn"
I20240502 08:17:58.723527 14034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c6b8f0f2531f40e9932bec32d4bd4988"
format_stamp: "Formatted at 2024-05-02 08:17:58 on dist-test-slave-x3qn"
I20240502 08:17:58.728761 14034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:17:58.732120 14048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:58.733698 14034 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:17:58.733896 14034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c6b8f0f2531f40e9932bec32d4bd4988"
format_stamp: "Formatted at 2024-05-02 08:17:58 on dist-test-slave-x3qn"
I20240502 08:17:58.734150 14034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:58.759692 14034 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:58.760558 14034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:58.760848 14034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:58.778398 14034 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.126:35417
I20240502 08:17:58.778410 14099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.126:35417 every 8 connection(s)
I20240502 08:17:58.779932 14034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:17:58.783567 14100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:17:58.788965  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14034
I20240502 08:17:58.789155  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:17:58.793314 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988: Bootstrap starting.
I20240502 08:17:58.796442 14100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988: Neither blocks nor log segments found. Creating new log.
I20240502 08:17:58.797505 14100 log.cc:824] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988: Log is configured to *not* fsync() on all Append() calls
I20240502 08:17:58.800647 14100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988: No bootstrap required, opened a new log
I20240502 08:17:58.808058 14100 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35417 } }
I20240502 08:17:58.808387 14100 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:17:58.808463 14100 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b8f0f2531f40e9932bec32d4bd4988, State: Initialized, Role: FOLLOWER
I20240502 08:17:58.809015 14100 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35417 } }
I20240502 08:17:58.809202 14100 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:17:58.809293 14100 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:17:58.809412 14100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:17:58.811467 14100 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35417 } }
I20240502 08:17:58.811906 14100 leader_election.cc:302] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6b8f0f2531f40e9932bec32d4bd4988; no voters: 
I20240502 08:17:58.812475 14100 leader_election.cc:288] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:17:58.812604 14105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:17:58.813256 14105 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [term 1 LEADER]: Becoming Leader. State: Replica: c6b8f0f2531f40e9932bec32d4bd4988, State: Running, Role: LEADER
I20240502 08:17:58.813866 14105 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35417 } }
I20240502 08:17:58.814477 14100 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:17:58.817258 14106 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35417 } } }
I20240502 08:17:58.817251 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6b8f0f2531f40e9932bec32d4bd4988. Latest consensus state: current_term: 1 leader_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6b8f0f2531f40e9932bec32d4bd4988" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 35417 } } }
I20240502 08:17:58.817763 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:58.817764 14106 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988 [sys.catalog]: This master's current role is: LEADER
I20240502 08:17:58.821435 14113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:17:58.826871 14113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:17:58.835851 14113 catalog_manager.cc:1347] Generated new cluster ID: dd7ee66a660b4ee8a5b0851f53f3211c
I20240502 08:17:58.835990 14113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:17:58.860708 14113 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:17:58.862537 14113 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:17:58.877805 14113 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c6b8f0f2531f40e9932bec32d4bd4988: Generated new TSK 0
I20240502 08:17:58.878803 14113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:17:58.893147  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.65:0
--local_ip_for_outbound_sockets=127.6.151.65
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:17:59.257019 14124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:59.257329 14124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:59.257421 14124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:59.265069 14124 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:59.265201 14124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:59.265326 14124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.65
I20240502 08:17:59.274556 14124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.65
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:59.275892 14124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:59.277662 14124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:59.289866 14130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:59.290022 14132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:59.290448 14129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:59.291061 14124 server_base.cc:1034] running on GCE node
I20240502 08:17:59.291647 14124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:59.292577 14124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:59.293771 14124 hybrid_clock.cc:648] HybridClock initialized: now 1714637879293712 us; error 51 us; skew 500 ppm
I20240502 08:17:59.294054 14124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:59.296293 14124 webserver.cc:460] Webserver started at http://127.6.151.65:33257/ using document root <none> and password file <none>
I20240502 08:17:59.296977 14124 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:59.297067 14124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:59.297343 14124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:59.300004 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
format_stamp: "Formatted at 2024-05-02 08:17:59 on dist-test-slave-x3qn"
I20240502 08:17:59.300706 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
format_stamp: "Formatted at 2024-05-02 08:17:59 on dist-test-slave-x3qn"
I20240502 08:17:59.307410 14124 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.007s	sys 0.000s
I20240502 08:17:59.310839 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:59.312208 14124 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 08:17:59.312348 14124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
format_stamp: "Formatted at 2024-05-02 08:17:59 on dist-test-slave-x3qn"
I20240502 08:17:59.312518 14124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:59.344291 14124 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:59.345221 14124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:59.345485 14124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:59.346494 14124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:59.347961 14124 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:59.348049 14124 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:59.348131 14124 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:59.348186 14124 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:59.379419 14124 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.65:43987
I20240502 08:17:59.379477 14250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.65:43987 every 8 connection(s)
I20240502 08:17:59.380864 14124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:17:59.381356  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14124
I20240502 08:17:59.381541  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:17:59.385056  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.66:0
--local_ip_for_outbound_sockets=127.6.151.66
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:59.394135 14251 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:17:59.394451 14251 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:59.395326 14251 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:17:59.397598 14065 ts_manager.cc:194] Registered new tserver with Master: b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:17:59.399875 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.65:57381
W20240502 08:17:59.724721 14255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:17:59.725031 14255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:17:59.725119 14255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:17:59.732784 14255 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:17:59.732915 14255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:17:59.733042 14255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.66
I20240502 08:17:59.743798 14255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.66
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:17:59.745093 14255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:17:59.746829 14255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:17:59.759178 14261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:59.759411 14263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:17:59.759719 14260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:17:59.760416 14255 server_base.cc:1034] running on GCE node
I20240502 08:17:59.761103 14255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:17:59.762091 14255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:17:59.763315 14255 hybrid_clock.cc:648] HybridClock initialized: now 1714637879763254 us; error 47 us; skew 500 ppm
I20240502 08:17:59.763664 14255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:17:59.766088 14255 webserver.cc:460] Webserver started at http://127.6.151.66:44879/ using document root <none> and password file <none>
I20240502 08:17:59.766952 14255 fs_manager.cc:359] Metadata directory not provided
I20240502 08:17:59.767072 14255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:17:59.767396 14255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:17:59.770059 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
format_stamp: "Formatted at 2024-05-02 08:17:59 on dist-test-slave-x3qn"
I20240502 08:17:59.770900 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
format_stamp: "Formatted at 2024-05-02 08:17:59 on dist-test-slave-x3qn"
I20240502 08:17:59.776124 14255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.003s
I20240502 08:17:59.779716 14269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:59.781234 14255 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 08:17:59.781411 14255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
format_stamp: "Formatted at 2024-05-02 08:17:59 on dist-test-slave-x3qn"
I20240502 08:17:59.781615 14255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:17:59.806265 14255 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:17:59.807163 14255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:17:59.807404 14255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:17:59.808313 14255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:17:59.809754 14255 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:17:59.809844 14255 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:59.809927 14255 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:17:59.810012 14255 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:17:59.841019 14255 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.66:44269
I20240502 08:17:59.841068 14381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.66:44269 every 8 connection(s)
I20240502 08:17:59.842394 14255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:17:59.848670  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14255
I20240502 08:17:59.848852  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:17:59.851977  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:0
--local_ip_for_outbound_sockets=127.6.151.67
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:17:59.855500 14382 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:17:59.855805 14382 heartbeater.cc:464] Registering TS with master...
I20240502 08:17:59.856611 14382 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:17:59.858028 14065 ts_manager.cc:194] Registered new tserver with Master: 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:17:59.858850 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.66:47089
W20240502 08:18:00.195804 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:00.196100 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:00.196192 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:00.203753 14385 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:00.203881 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:00.204020 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:18:00.213146 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:00.214448 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:00.216284 14385 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:00.228255 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:00.228506 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:00.228686 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:00.229409 14385 server_base.cc:1034] running on GCE node
I20240502 08:18:00.230046 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:00.231114 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:00.232352 14385 hybrid_clock.cc:648] HybridClock initialized: now 1714637880232313 us; error 31 us; skew 500 ppm
I20240502 08:18:00.232729 14385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:00.235190 14385 webserver.cc:460] Webserver started at http://127.6.151.67:38925/ using document root <none> and password file <none>
I20240502 08:18:00.235963 14385 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:00.236088 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:00.236414 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:00.241065 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
format_stamp: "Formatted at 2024-05-02 08:18:00 on dist-test-slave-x3qn"
I20240502 08:18:00.241955 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
format_stamp: "Formatted at 2024-05-02 08:18:00 on dist-test-slave-x3qn"
I20240502 08:18:00.247637 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:18:00.251111 14399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:00.252458 14385 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:18:00.252621 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
format_stamp: "Formatted at 2024-05-02 08:18:00 on dist-test-slave-x3qn"
I20240502 08:18:00.252823 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:00.268030 14385 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:00.268906 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:00.269184 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:00.270143 14385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:00.271728 14385 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:00.271827 14385 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:00.271939 14385 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:00.272009 14385 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:00.304831 14385 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:39913
I20240502 08:18:00.304889 14513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:39913 every 8 connection(s)
I20240502 08:18:00.306259 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:18:00.314960  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14385
I20240502 08:18:00.315107  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:18:00.318789  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:0
--local_ip_for_outbound_sockets=127.6.151.68
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:00.319114 14514 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:00.319422 14514 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:00.320293 14514 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:00.321830 14065 ts_manager.cc:194] Registered new tserver with Master: 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:00.322994 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:41635
I20240502 08:18:00.402935 14251 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
W20240502 08:18:00.660426 14517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:00.660743 14517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:00.660900 14517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:00.668457 14517 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:00.668613 14517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:00.668782 14517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:18:00.678125 14517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:00.679544 14517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:00.681317 14517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:00.693576 14525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:00.693506 14522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:00.693859 14523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:00.694514 14517 server_base.cc:1034] running on GCE node
I20240502 08:18:00.695130 14517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:00.696058 14517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:00.697258 14517 hybrid_clock.cc:648] HybridClock initialized: now 1714637880697219 us; error 36 us; skew 500 ppm
I20240502 08:18:00.697577 14517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:00.699863 14517 webserver.cc:460] Webserver started at http://127.6.151.68:36177/ using document root <none> and password file <none>
I20240502 08:18:00.700579 14517 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:00.700698 14517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:00.701022 14517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:00.703637 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e79a5862de19490b89a9ea97860d776b"
format_stamp: "Formatted at 2024-05-02 08:18:00 on dist-test-slave-x3qn"
I20240502 08:18:00.704402 14517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e79a5862de19490b89a9ea97860d776b"
format_stamp: "Formatted at 2024-05-02 08:18:00 on dist-test-slave-x3qn"
I20240502 08:18:00.709568 14517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 08:18:00.712967 14532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:00.714363 14517 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:18:00.714536 14517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e79a5862de19490b89a9ea97860d776b"
format_stamp: "Formatted at 2024-05-02 08:18:00 on dist-test-slave-x3qn"
I20240502 08:18:00.714800 14517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:00.731999 14517 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:00.732861 14517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:00.733139 14517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:00.734145 14517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:00.735690 14517 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:00.735797 14517 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:00.735915 14517 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:00.735988 14517 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:00.769881 14517 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:42587
I20240502 08:18:00.769937 14644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:42587 every 8 connection(s)
I20240502 08:18:00.771260 14517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:18:00.780572  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14517
I20240502 08:18:00.780745  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:18:00.784235 14645 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:00.784540 14645 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:00.784641  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:0
--local_ip_for_outbound_sockets=127.6.151.69
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:00.785470 14645 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:00.786898 14065 ts_manager.cc:194] Registered new tserver with Master: e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:00.787730 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:53953
I20240502 08:18:00.861299 14382 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
W20240502 08:18:01.133899 14648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:01.134215 14648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:01.134346 14648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:01.141975 14648 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:01.142115 14648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:01.142289 14648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:18:01.151484 14648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:01.152854 14648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:01.154556 14648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:01.166776 14653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:01.167014 14654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:01.167217 14656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:01.167948 14648 server_base.cc:1034] running on GCE node
I20240502 08:18:01.168565 14648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:01.169509 14648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:01.170763 14648 hybrid_clock.cc:648] HybridClock initialized: now 1714637881170698 us; error 55 us; skew 500 ppm
I20240502 08:18:01.171068 14648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:01.173449 14648 webserver.cc:460] Webserver started at http://127.6.151.69:38959/ using document root <none> and password file <none>
I20240502 08:18:01.174189 14648 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:01.174304 14648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:01.174630 14648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:01.177430 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:01.178232 14648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:01.183667 14648 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240502 08:18:01.187615 14662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:01.189193 14648 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:18:01.189404 14648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:01.189647 14648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:01.203584 14648 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:01.204468 14648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:01.204759 14648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:01.205745 14648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:01.207271 14648 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:01.207370 14648 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:01.207487 14648 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:01.207559 14648 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:01.244191 14648 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:43257
I20240502 08:18:01.244261 14775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:43257 every 8 connection(s)
I20240502 08:18:01.245534 14648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:18:01.248111  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14648
I20240502 08:18:01.248298  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:18:01.251323  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.70:0
--local_ip_for_outbound_sockets=127.6.151.70
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:01.259286 14776 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:01.259613 14776 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:01.260460 14776 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:01.261935 14065 ts_manager.cc:194] Registered new tserver with Master: f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257)
I20240502 08:18:01.262763 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:54095
I20240502 08:18:01.325533 14514 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
W20240502 08:18:01.607178 14779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:01.607473 14779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:01.607568 14779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:01.615193 14779 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:01.615327 14779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:01.615448 14779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.70
I20240502 08:18:01.625444 14779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.70
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:01.626834 14779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:01.628515 14779 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:01.640450 14784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:01.640697 14785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:01.641016 14787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:01.641736 14779 server_base.cc:1034] running on GCE node
I20240502 08:18:01.642320 14779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:01.643299 14779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:01.644491 14779 hybrid_clock.cc:648] HybridClock initialized: now 1714637881644431 us; error 54 us; skew 500 ppm
I20240502 08:18:01.644793 14779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:01.647194 14779 webserver.cc:460] Webserver started at http://127.6.151.70:40841/ using document root <none> and password file <none>
I20240502 08:18:01.647938 14779 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:01.648042 14779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:01.648367 14779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:01.651047 14779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "5b84b46faddb47fcae86cc5f3f478bf1"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:01.651847 14779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "5b84b46faddb47fcae86cc5f3f478bf1"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:01.657073 14779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:18:01.660485 14793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:01.661872 14779 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:01.662039 14779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "5b84b46faddb47fcae86cc5f3f478bf1"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:01.662238 14779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:01.702229 14779 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:01.703159 14779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:01.703439 14779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:01.704404 14779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:01.705883 14779 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:01.705986 14779 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:01.706107 14779 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:01.706188 14779 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:01.739262 14779 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.70:45703
I20240502 08:18:01.739331 14906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.70:45703 every 8 connection(s)
I20240502 08:18:01.740558 14779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:18:01.745935  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14779
I20240502 08:18:01.746127  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:18:01.749644  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.71:0
--local_ip_for_outbound_sockets=127.6.151.71
--webserver_interface=127.6.151.71
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:01.755030 14907 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:01.755333 14907 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:01.756175 14907 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:01.757717 14065 ts_manager.cc:194] Registered new tserver with Master: 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703)
I20240502 08:18:01.758715 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.70:34181
I20240502 08:18:01.790295 14645 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
W20240502 08:18:02.100382 14910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:02.100677 14910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:02.100821 14910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:02.108353 14910 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:02.108484 14910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:02.108672 14910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.71
I20240502 08:18:02.117985 14910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.151.71
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.71
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:02.119405 14910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:02.121203 14910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:02.133536 14915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:02.133771 14916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:02.134030 14918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:02.134315 14910 server_base.cc:1034] running on GCE node
I20240502 08:18:02.134975 14910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:02.135881 14910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:02.137082 14910 hybrid_clock.cc:648] HybridClock initialized: now 1714637882137040 us; error 37 us; skew 500 ppm
I20240502 08:18:02.137388 14910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:02.139616 14910 webserver.cc:460] Webserver started at http://127.6.151.71:37309/ using document root <none> and password file <none>
I20240502 08:18:02.140327 14910 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:02.140439 14910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:02.140764 14910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:02.143447 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b3dfd9d6472342689afd86203f7afe5f"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:02.144217 14910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b3dfd9d6472342689afd86203f7afe5f"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:02.149616 14910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:18:02.153185 14924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:02.154578 14910 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:18:02.154778 14910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b3dfd9d6472342689afd86203f7afe5f"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:02.154983 14910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:02.174328 14910 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:02.175262 14910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:02.175536 14910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:02.176553 14910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:02.178103 14910 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:02.178205 14910 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:02.178325 14910 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:02.178396 14910 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:02.210460 14910 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.71:39897
I20240502 08:18:02.210513 15037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.71:39897 every 8 connection(s)
I20240502 08:18:02.211819 14910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:18:02.213016  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 14910
I20240502 08:18:02.213200  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 08:18:02.216881  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.72:0
--local_ip_for_outbound_sockets=127.6.151.72
--webserver_interface=127.6.151.72
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:02.226699 15038 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:02.227047 15038 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:02.228109 15038 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:02.229763 14065 ts_manager.cc:194] Registered new tserver with Master: b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897)
I20240502 08:18:02.230583 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.71:57879
I20240502 08:18:02.265789 14776 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
W20240502 08:18:02.573559 15041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:02.573859 15041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:02.573961 15041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:02.581526 15041 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:02.581656 15041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:02.581772 15041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.72
I20240502 08:18:02.590986 15041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.151.72
--webserver_port=0
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.72
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:02.592336 15041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:02.594090 15041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:02.606443 15047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:02.606659 15046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:02.607017 15049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:02.607605 15041 server_base.cc:1034] running on GCE node
I20240502 08:18:02.608192 15041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:02.609141 15041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:02.610358 15041 hybrid_clock.cc:648] HybridClock initialized: now 1714637882610304 us; error 51 us; skew 500 ppm
I20240502 08:18:02.610702 15041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:02.612957 15041 webserver.cc:460] Webserver started at http://127.6.151.72:38051/ using document root <none> and password file <none>
I20240502 08:18:02.613691 15041 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:02.613811 15041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:02.614127 15041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:02.616874 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "42116b63c31a436ca060c34691780d04"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:02.617682 15041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "42116b63c31a436ca060c34691780d04"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:02.623553 15041 fs_manager.cc:691] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20240502 08:18:02.628371 15055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:02.630179 15041 fs_manager.cc:725] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 08:18:02.630393 15041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "42116b63c31a436ca060c34691780d04"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:02.630594 15041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:02.648308 15041 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:02.649171 15041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:02.649453 15041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:02.650430 15041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:02.651955 15041 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:02.652060 15041 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:02.652176 15041 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:02.652256 15041 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:02.684412 15041 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.72:44955
I20240502 08:18:02.684464 15168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.72:44955 every 8 connection(s)
I20240502 08:18:02.685765 15041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:18:02.691617  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 15041
I20240502 08:18:02.691810  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 08:18:02.698454 15169 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:02.698766 15169 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:02.699604 15169 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:02.700932 14065 ts_manager.cc:194] Registered new tserver with Master: 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955)
I20240502 08:18:02.701712 14065 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.72:47241
I20240502 08:18:02.707760  6749 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:18:02.725009  6749 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 08:18:02.725135  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14648
I20240502 08:18:02.733363 14771 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:02.761289 14907 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
I20240502 08:18:02.815891  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14648
I20240502 08:18:02.833204  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14779
I20240502 08:18:02.841495 14902 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:02.924333  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14779
I20240502 08:18:02.943106  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14910
I20240502 08:18:02.951190 15033 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:03.034112  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14910
I20240502 08:18:03.051563  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 15041
I20240502 08:18:03.059945 15164 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:03.140967  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 15041
I20240502 08:18:06.910243 14065 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:06.948963 14186 tablet_service.cc:1467] Processing CreateTablet for tablet 6272b2f3a9504e01832cb31f5b322980 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.949105 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 96365c4348464db59463182aea8c1c23 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.950590 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96365c4348464db59463182aea8c1c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.951721 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 96365c4348464db59463182aea8c1c23 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.952759 14579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96365c4348464db59463182aea8c1c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.950174 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 6272b2f3a9504e01832cb31f5b322980 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.954489 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6272b2f3a9504e01832cb31f5b322980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.955646 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 6272b2f3a9504e01832cb31f5b322980 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.956712 14317 tablet_service.cc:1467] Processing CreateTablet for tablet f0bcdf713f124d46a22ddfc2873cd742 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.957105 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6272b2f3a9504e01832cb31f5b322980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.957201 14448 tablet_service.cc:1467] Processing CreateTablet for tablet f0bcdf713f124d46a22ddfc2873cd742 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.957803 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bcdf713f124d46a22ddfc2873cd742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.957859 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bcdf713f124d46a22ddfc2873cd742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.963179 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6272b2f3a9504e01832cb31f5b322980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.955631 14316 tablet_service.cc:1467] Processing CreateTablet for tablet 96365c4348464db59463182aea8c1c23 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.967236 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96365c4348464db59463182aea8c1c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.949319 14185 tablet_service.cc:1467] Processing CreateTablet for tablet f0bcdf713f124d46a22ddfc2873cd742 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:06.967729 14185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bcdf713f124d46a22ddfc2873cd742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:06.968734 15198 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:06.972517 15198 tablet_bootstrap.cc:654] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.975443 15198 log.cc:824] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:06.976996 15199 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:06.981559 15199 tablet_bootstrap.cc:654] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:06.983155 15199 log.cc:824] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:06.987021 15198 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:06.987479 15198 ts_tablet_manager.cc:1367] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.006s
I20240502 08:18:06.992456 15199 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:06.992874 15199 ts_tablet_manager.cc:1367] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.007s
I20240502 08:18:06.997092 15198 raft_consensus.cc:348] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:06.997787 15198 raft_consensus.cc:374] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:06.999524 15202 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:06.999670 15198 raft_consensus.cc:729] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.000473 15198 consensus_queue.cc:260] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.002349 15198 ts_tablet_manager.cc:1398] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.015s	user 0.012s	sys 0.000s
I20240502 08:18:07.002864 15198 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:07.006737 15203 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:07.007014 15198 tablet_bootstrap.cc:654] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.007176 15202 tablet_bootstrap.cc:654] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.007778 15199 raft_consensus.cc:348] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.008302 15199 raft_consensus.cc:374] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.008450 15199 raft_consensus.cc:729] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.008582 15202 log.cc:824] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:07.009312 15199 consensus_queue.cc:260] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.010779 15203 tablet_bootstrap.cc:654] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.012732 15203 log.cc:824] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:07.014959 15199 ts_tablet_manager.cc:1398] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.022s	user 0.013s	sys 0.002s
I20240502 08:18:07.015503 15199 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:07.017699 15199 tablet_bootstrap.cc:654] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.018416 15198 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:07.018739 15198 ts_tablet_manager.cc:1367] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.003s
I20240502 08:18:07.020051 15198 raft_consensus.cc:348] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.020380 15198 raft_consensus.cc:374] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.020525 15198 raft_consensus.cc:729] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.020884 15198 consensus_queue.cc:260] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.021314 15202 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:07.021610 15202 ts_tablet_manager.cc:1367] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.009s
I20240502 08:18:07.023679 15203 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:07.023792 15198 ts_tablet_manager.cc:1398] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:18:07.023819 15199 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:07.023955 15199 ts_tablet_manager.cc:1367] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.003s
I20240502 08:18:07.023973 15203 ts_tablet_manager.cc:1367] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.018s	user 0.005s	sys 0.005s
I20240502 08:18:07.024969 15199 raft_consensus.cc:348] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.025192 15199 raft_consensus.cc:374] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.025285 15199 raft_consensus.cc:729] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.025501 15199 consensus_queue.cc:260] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.026037 15199 ts_tablet_manager.cc:1398] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240502 08:18:07.027101 14646 tablet.cc:2369] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:07.030216 15202 raft_consensus.cc:348] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.030552 15202 raft_consensus.cc:374] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.030676 15202 raft_consensus.cc:729] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.031294 15202 consensus_queue.cc:260] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.032469 15203 raft_consensus.cc:348] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.032824 15202 ts_tablet_manager.cc:1398] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20240502 08:18:07.032889 15203 raft_consensus.cc:374] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.033001 15203 raft_consensus.cc:729] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.033289 15202 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:07.033766 15203 consensus_queue.cc:260] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.034988 15202 tablet_bootstrap.cc:654] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.035758 15203 ts_tablet_manager.cc:1398] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
I20240502 08:18:07.036176 15203 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:07.036969 15202 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:07.037112 15202 ts_tablet_manager.cc:1367] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:18:07.037827 15203 tablet_bootstrap.cc:654] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.037963 15202 raft_consensus.cc:348] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.038527 15202 raft_consensus.cc:374] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.038621 15202 raft_consensus.cc:729] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.038887 15202 consensus_queue.cc:260] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.039561 15202 ts_tablet_manager.cc:1398] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:18:07.039850 15202 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:07.041069 15203 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:07.041218 15203 ts_tablet_manager.cc:1367] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20240502 08:18:07.041452 15202 tablet_bootstrap.cc:654] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:07.042268 15203 raft_consensus.cc:348] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.042531 15203 raft_consensus.cc:374] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.042611 15203 raft_consensus.cc:729] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.042881 15203 consensus_queue.cc:260] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.043606 15202 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:07.043821 15202 ts_tablet_manager.cc:1367] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.004s	user 0.002s	sys 0.001s
I20240502 08:18:07.044086 15203 ts_tablet_manager.cc:1398] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:07.044464 15202 raft_consensus.cc:348] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.044636 15202 raft_consensus.cc:374] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:07.044729 15202 raft_consensus.cc:729] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:07.045017 15202 consensus_queue.cc:260] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.045540 15202 ts_tablet_manager.cc:1398] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
W20240502 08:18:07.062722 14515 tablet.cc:2369] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:18:07.099539 14383 tablet.cc:2369] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:07.118777 15207 raft_consensus.cc:482] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.119053 15207 raft_consensus.cc:504] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.120926 15207 leader_election.cc:288] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:07.128260 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6272b2f3a9504e01832cb31f5b322980" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:07.128240 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6272b2f3a9504e01832cb31f5b322980" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:07.128751 14600 raft_consensus.cc:2454] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:07.128755 14206 raft_consensus.cc:2454] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:07.129338 14400 leader_election.cc:302] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:07.129747 15207 raft_consensus.cc:2781] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.129875 15207 raft_consensus.cc:482] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.129945 15207 raft_consensus.cc:3037] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.132047 15207 raft_consensus.cc:504] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.132633 15207 leader_election.cc:288] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Requested vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:07.133232 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6272b2f3a9504e01832cb31f5b322980" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:07.133245 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6272b2f3a9504e01832cb31f5b322980" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:07.133464 14206 raft_consensus.cc:3037] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.133462 14600 raft_consensus.cc:3037] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.136667 14600 raft_consensus.cc:2454] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:07.136668 14206 raft_consensus.cc:2454] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:07.137144 14402 leader_election.cc:302] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:07.137477 15207 raft_consensus.cc:2781] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.137977 15207 raft_consensus.cc:686] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:07.138512 15207 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
W20240502 08:18:07.138679 14252 tablet.cc:2369] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:18:07.143254 14065 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67). New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:07.240931 15211 raft_consensus.cc:482] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.241220 15211 raft_consensus.cc:504] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.243029 15211 leader_election.cc:288] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:07.249922 15210 raft_consensus.cc:482] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:07.250161 15210 raft_consensus.cc:504] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.250293 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96365c4348464db59463182aea8c1c23" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:07.250499 14600 raft_consensus.cc:2454] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:07.250716 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96365c4348464db59463182aea8c1c23" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:07.250959 14206 raft_consensus.cc:2454] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:07.251165 14272 leader_election.cc:302] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:07.251613 15211 raft_consensus.cc:2781] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.251771 15211 raft_consensus.cc:482] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.251878 15211 raft_consensus.cc:3037] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.252382 15210 leader_election.cc:288] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:07.254451 15211 raft_consensus.cc:504] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.255743 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96365c4348464db59463182aea8c1c23" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:07.255970 14600 raft_consensus.cc:3037] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.256717 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96365c4348464db59463182aea8c1c23" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:07.256928 14206 raft_consensus.cc:3037] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.258327 14600 raft_consensus.cc:2454] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:07.258852 14272 leader_election.cc:302] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:07.259368 15211 leader_election.cc:288] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Requested vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:07.259426 14206 raft_consensus.cc:2454] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:07.259598 15211 raft_consensus.cc:2781] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.259773 15211 raft_consensus.cc:686] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:07.260752 15211 consensus_queue.cc:237] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:07.262140 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bcdf713f124d46a22ddfc2873cd742" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:07.262511 14469 raft_consensus.cc:2454] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:07.263065 14141 leader_election.cc:302] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:07.263437 15210 raft_consensus.cc:2781] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:07.263572 15210 raft_consensus.cc:482] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:07.263689 15210 raft_consensus.cc:3037] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.264286 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bcdf713f124d46a22ddfc2873cd742" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:07.264665 14337 raft_consensus.cc:2454] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:07.266156 15210 raft_consensus.cc:504] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.266783 15210 leader_election.cc:288] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:07.267270 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bcdf713f124d46a22ddfc2873cd742" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:07.267278 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0bcdf713f124d46a22ddfc2873cd742" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:07.267467 14469 raft_consensus.cc:3037] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.267470 14337 raft_consensus.cc:3037] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:07.270536 14337 raft_consensus.cc:2454] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:07.270530 14469 raft_consensus.cc:2454] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:07.270989 14141 leader_election.cc:302] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:07.271334 15210 raft_consensus.cc:2781] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:07.271904 15210 raft_consensus.cc:686] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:07.272802 15210 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:07.273528 14065 catalog_manager.cc:5484] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:07.277310 14064 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:07.651513 15227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:07.651888 15227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:07.659597 15227 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:07.671690 15231 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.672096 15210 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.683693 15210 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.686412 15211 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.700603 15223 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:07.702224 15245 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 318.851 ms
  time per row: 0.159426 ms
I20240502 08:18:08.154971 14064 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:08.164064 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 930e3af93722422fb9237fd9b623ceac (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.164229 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 930e3af93722422fb9237fd9b623ceac (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.164618 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930e3af93722422fb9237fd9b623ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.164727 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930e3af93722422fb9237fd9b623ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.165752 14448 tablet_service.cc:1467] Processing CreateTablet for tablet d33a9164ef974a8caae83f3c22b87a51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.165927 14447 tablet_service.cc:1467] Processing CreateTablet for tablet 44f747cd88e04d23951f351b1578bbb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.166177 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a9164ef974a8caae83f3c22b87a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.166462 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f747cd88e04d23951f351b1578bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.167491 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 930e3af93722422fb9237fd9b623ceac (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.167580 14579 tablet_service.cc:1467] Processing CreateTablet for tablet d33a9164ef974a8caae83f3c22b87a51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.167950 14579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a9164ef974a8caae83f3c22b87a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.168217 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930e3af93722422fb9237fd9b623ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.173592 14315 tablet_service.cc:1467] Processing CreateTablet for tablet 44f747cd88e04d23951f351b1578bbb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.173591 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 44f747cd88e04d23951f351b1578bbb8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.174108 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f747cd88e04d23951f351b1578bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.174103 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f747cd88e04d23951f351b1578bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.179163 14316 tablet_service.cc:1467] Processing CreateTablet for tablet d33a9164ef974a8caae83f3c22b87a51 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:08.179628 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a9164ef974a8caae83f3c22b87a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:08.185024 15300 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:08.186587 15299 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:08.188657 15300 tablet_bootstrap.cc:654] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.188760 15299 tablet_bootstrap.cc:654] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.191068 15303 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:08.195868 15303 tablet_bootstrap.cc:654] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.197135 15299 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:08.197366 15299 ts_tablet_manager.cc:1367] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
I20240502 08:18:08.198335 15299 raft_consensus.cc:348] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.198695 15301 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:08.198743 15299 raft_consensus.cc:374] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.198889 15299 raft_consensus.cc:729] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.199237 15299 consensus_queue.cc:260] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.200731 15299 ts_tablet_manager.cc:1398] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:18:08.201063 15299 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:08.201467 15303 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:08.201711 15303 ts_tablet_manager.cc:1367] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.011s	user 0.002s	sys 0.003s
I20240502 08:18:08.202893 15299 tablet_bootstrap.cc:654] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.202800 15303 raft_consensus.cc:348] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.203096 15303 raft_consensus.cc:374] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.203262 15303 raft_consensus.cc:729] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.203807 15303 consensus_queue.cc:260] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.204592 15303 ts_tablet_manager.cc:1398] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:18:08.205515 15301 tablet_bootstrap.cc:654] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.207893 15300 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:08.208141 15300 ts_tablet_manager.cc:1367] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.011s
I20240502 08:18:08.208158 15299 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:08.208317 15299 ts_tablet_manager.cc:1367] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.007s	user 0.000s	sys 0.004s
I20240502 08:18:08.208479 15301 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:08.208700 15301 ts_tablet_manager.cc:1367] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.004s
I20240502 08:18:08.209177 15299 raft_consensus.cc:348] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.209453 15299 raft_consensus.cc:374] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.209573 15299 raft_consensus.cc:729] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.209612 15300 raft_consensus.cc:348] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.209867 15300 raft_consensus.cc:374] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.209616 15301 raft_consensus.cc:348] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.210000 15300 raft_consensus.cc:729] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.209951 15299 consensus_queue.cc:260] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.210273 15301 raft_consensus.cc:374] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.210253 15300 consensus_queue.cc:260] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.210599 15299 ts_tablet_manager.cc:1398] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:18:08.210882 15300 ts_tablet_manager.cc:1398] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:18:08.211006 15299 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:08.211243 15300 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:08.210429 15301 raft_consensus.cc:729] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.212061 15301 consensus_queue.cc:260] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.212694 15299 tablet_bootstrap.cc:654] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.212803 15301 ts_tablet_manager.cc:1398] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 08:18:08.212865 15300 tablet_bootstrap.cc:654] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.213243 15301 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:08.215456 15301 tablet_bootstrap.cc:654] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.224217 15301 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:08.224558 15301 ts_tablet_manager.cc:1367] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.011s	user 0.008s	sys 0.002s
I20240502 08:18:08.225664 15301 raft_consensus.cc:348] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.226104 15301 raft_consensus.cc:374] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.226253 15301 raft_consensus.cc:729] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.226712 15301 consensus_queue.cc:260] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.227633 15301 ts_tablet_manager.cc:1398] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:08.228106 15301 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:08.230481 15301 tablet_bootstrap.cc:654] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:08.233878 15300 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:08.234148 15300 ts_tablet_manager.cc:1367] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.007s
I20240502 08:18:08.234396 15299 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:08.234552 15299 ts_tablet_manager.cc:1367] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.012s
I20240502 08:18:08.234925 15301 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:08.235082 15301 ts_tablet_manager.cc:1367] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 08:18:08.235050 15300 raft_consensus.cc:348] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.235319 15300 raft_consensus.cc:374] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.235248 15299 raft_consensus.cc:348] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.235401 15300 raft_consensus.cc:729] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.235466 15299 raft_consensus.cc:374] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.235565 15299 raft_consensus.cc:729] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.235706 15300 consensus_queue.cc:260] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.235808 15299 consensus_queue.cc:260] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.235857 15301 raft_consensus.cc:348] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.236249 15300 ts_tablet_manager.cc:1398] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:18:08.236863 15299 ts_tablet_manager.cc:1398] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:08.238652 15301 raft_consensus.cc:374] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:08.238811 15301 raft_consensus.cc:729] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:08.239078 15301 consensus_queue.cc:260] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.240662 15301 ts_tablet_manager.cc:1398] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:18:08.242775 15252 raft_consensus.cc:482] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.242921 15252 raft_consensus.cc:504] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.244495 15252 leader_election.cc:288] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:08.251274 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930e3af93722422fb9237fd9b623ceac" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:08.251547 14337 raft_consensus.cc:2454] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:08.252076 14535 leader_election.cc:302] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:08.252185 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930e3af93722422fb9237fd9b623ceac" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:08.252435 14469 raft_consensus.cc:2454] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:08.252516 15252 raft_consensus.cc:2781] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.252642 15252 raft_consensus.cc:482] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.252743 15252 raft_consensus.cc:3037] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.254783 15252 raft_consensus.cc:504] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.255350 15252 leader_election.cc:288] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:08.255817 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930e3af93722422fb9237fd9b623ceac" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:08.255911 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930e3af93722422fb9237fd9b623ceac" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:08.256012 14337 raft_consensus.cc:3037] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.256155 14469 raft_consensus.cc:3037] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.258298 14469 raft_consensus.cc:2454] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:08.258538 14337 raft_consensus.cc:2454] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:08.258735 14535 leader_election.cc:302] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:08.259059 15252 raft_consensus.cc:2781] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.259474 15252 raft_consensus.cc:686] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:08.259950 15252 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.263396 14064 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 0 to 1, leader changed from <none> to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:08.276543 15231 raft_consensus.cc:482] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.276762 15231 raft_consensus.cc:504] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.277529 15231 leader_election.cc:288] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:08.278158 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f747cd88e04d23951f351b1578bbb8" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:08.278394 14206 raft_consensus.cc:2454] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:08.278862 14400 leader_election.cc:302] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:08.279155 15231 raft_consensus.cc:2781] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.279256 15231 raft_consensus.cc:482] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.279372 15231 raft_consensus.cc:3037] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.280277 15252 raft_consensus.cc:482] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:08.280483 15252 raft_consensus.cc:504] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.281167 15252 leader_election.cc:288] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:08.281572 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33a9164ef974a8caae83f3c22b87a51" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:08.281661 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33a9164ef974a8caae83f3c22b87a51" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:08.281816 14337 raft_consensus.cc:2454] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:08.281894 14469 raft_consensus.cc:2454] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:08.282008 15231 raft_consensus.cc:504] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.282281 14535 leader_election.cc:302] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:08.282708 15252 raft_consensus.cc:2781] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:08.282917 15252 raft_consensus.cc:482] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:08.283046 15252 raft_consensus.cc:3037] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.283239 15231 leader_election.cc:288] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:08.283285 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f747cd88e04d23951f351b1578bbb8" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:08.283509 14206 raft_consensus.cc:3037] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.285611 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f747cd88e04d23951f351b1578bbb8" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:08.285745 14206 raft_consensus.cc:2454] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:08.285912 14337 raft_consensus.cc:2454] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:08.285732 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f747cd88e04d23951f351b1578bbb8" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:08.285878 15252 raft_consensus.cc:504] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.286345 14400 leader_election.cc:302] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:08.286774 15231 raft_consensus.cc:2781] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.286842 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33a9164ef974a8caae83f3c22b87a51" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:08.286993 15231 raft_consensus.cc:686] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:08.287067 14336 raft_consensus.cc:3037] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.287328 15252 leader_election.cc:288] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:08.287312 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d33a9164ef974a8caae83f3c22b87a51" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:08.287560 14469 raft_consensus.cc:3037] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.289175 14336 raft_consensus.cc:2454] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:08.289374 14469 raft_consensus.cc:2454] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:08.289590 14535 leader_election.cc:302] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:08.289589 15231 consensus_queue.cc:237] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:08.290069 15252 raft_consensus.cc:2781] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:08.290225 15252 raft_consensus.cc:686] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:08.290455 15252 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:08.292670 14064 catalog_manager.cc:5484] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67). New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:08.293715 14058 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 0 to 1, leader changed from <none> to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:08.685240 14337 raft_consensus.cc:3037] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:08.688234 15231 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.693017 15252 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.707964 15252 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.709151 15231 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.742774 15252 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:08.750111 15326 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:08.821216 15319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:08.821723 15319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:08.837867 15319 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 205.2 ms
  time per row: 0.1026 ms
I20240502 08:18:09.194823 14058 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:09.203809 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 0815524d0eb448438dae05f646f14580 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.203995 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 0815524d0eb448438dae05f646f14580 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.204290 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0815524d0eb448438dae05f646f14580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.204421 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0815524d0eb448438dae05f646f14580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.204352 14316 tablet_service.cc:1467] Processing CreateTablet for tablet 0815524d0eb448438dae05f646f14580 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.204742 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0815524d0eb448438dae05f646f14580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.205981 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 827a2fb0396441749a35977e2854bfb6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.206180 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 88890b0913c0462bade8208dfa0f64b7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.206413 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827a2fb0396441749a35977e2854bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.206530 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88890b0913c0462bade8208dfa0f64b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.207047 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 827a2fb0396441749a35977e2854bfb6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.207413 14579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827a2fb0396441749a35977e2854bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.211930 14186 tablet_service.cc:1467] Processing CreateTablet for tablet 88890b0913c0462bade8208dfa0f64b7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.212380 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88890b0913c0462bade8208dfa0f64b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.215198 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 88890b0913c0462bade8208dfa0f64b7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.215718 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88890b0913c0462bade8208dfa0f64b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.218824 14185 tablet_service.cc:1467] Processing CreateTablet for tablet 827a2fb0396441749a35977e2854bfb6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:09.219311 14185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827a2fb0396441749a35977e2854bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:09.224483 15383 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:09.227483 15383 tablet_bootstrap.cc:654] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.235244 15384 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:09.240008 15384 tablet_bootstrap.cc:654] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.241845 15385 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:09.242559 15382 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:09.244287 15384 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:09.244328 15383 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:09.244482 15385 tablet_bootstrap.cc:654] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.244547 15384 ts_tablet_manager.cc:1367] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.000s
I20240502 08:18:09.244587 15383 ts_tablet_manager.cc:1367] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.007s
I20240502 08:18:09.244614 15382 tablet_bootstrap.cc:654] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.245507 15384 raft_consensus.cc:348] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.245623 15383 raft_consensus.cc:348] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.245966 15384 raft_consensus.cc:374] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.246048 15383 raft_consensus.cc:374] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.246209 15383 raft_consensus.cc:729] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.246204 15384 raft_consensus.cc:729] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.246557 15383 consensus_queue.cc:260] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.247045 15384 consensus_queue.cc:260] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.247960 15383 ts_tablet_manager.cc:1398] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:09.248435 15384 ts_tablet_manager.cc:1398] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:18:09.248790 15384 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:09.250499 15384 tablet_bootstrap.cc:654] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.254526 15382 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:09.254706 15382 ts_tablet_manager.cc:1367] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.009s
I20240502 08:18:09.255240 15384 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:09.255409 15384 ts_tablet_manager.cc:1367] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 08:18:09.255568 15382 raft_consensus.cc:348] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.255851 15382 raft_consensus.cc:374] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.255954 15382 raft_consensus.cc:729] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.256189 15384 raft_consensus.cc:348] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.256206 15382 consensus_queue.cc:260] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.256456 15384 raft_consensus.cc:374] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.256574 15384 raft_consensus.cc:729] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.256958 15382 ts_tablet_manager.cc:1398] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:09.256850 15384 consensus_queue.cc:260] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.257261 15382 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:09.257978 15384 ts_tablet_manager.cc:1398] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:09.258376 15384 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:09.258930 15382 tablet_bootstrap.cc:654] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.260521 15385 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:09.260599 15384 tablet_bootstrap.cc:654] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.260702 15385 ts_tablet_manager.cc:1367] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.003s
I20240502 08:18:09.261597 15385 raft_consensus.cc:348] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.261849 15385 raft_consensus.cc:374] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.261976 15385 raft_consensus.cc:729] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.262260 15385 consensus_queue.cc:260] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.262864 15385 ts_tablet_manager.cc:1398] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 08:18:09.262950 15384 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:09.263005 15382 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:09.263134 15384 ts_tablet_manager.cc:1367] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:18:09.263202 15382 ts_tablet_manager.cc:1367] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.006s	user 0.003s	sys 0.000s
I20240502 08:18:09.263635 15385 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:09.263826 15384 raft_consensus.cc:348] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.263942 15382 raft_consensus.cc:348] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.264091 15384 raft_consensus.cc:374] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.264206 15382 raft_consensus.cc:374] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.264222 15384 raft_consensus.cc:729] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.264319 15382 raft_consensus.cc:729] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.264539 15384 consensus_queue.cc:260] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.264591 15382 consensus_queue.cc:260] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.265053 15384 ts_tablet_manager.cc:1398] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:18:09.265681 15382 ts_tablet_manager.cc:1398] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:09.266029 15382 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:09.266045 15385 tablet_bootstrap.cc:654] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.267943 15382 tablet_bootstrap.cc:654] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:09.272105 15382 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:09.272248 15382 ts_tablet_manager.cc:1367] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240502 08:18:09.272346 15385 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:09.272517 15385 ts_tablet_manager.cc:1367] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.003s
I20240502 08:18:09.272835 15382 raft_consensus.cc:348] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.273005 15382 raft_consensus.cc:374] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.273102 15382 raft_consensus.cc:729] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.273141 15385 raft_consensus.cc:348] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.273334 15385 raft_consensus.cc:374] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:09.273420 15385 raft_consensus.cc:729] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:09.273327 15382 consensus_queue.cc:260] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.273806 15382 ts_tablet_manager.cc:1398] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:18:09.273705 15385 consensus_queue.cc:260] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.274255 15385 ts_tablet_manager.cc:1398] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:18:09.294422 15223 raft_consensus.cc:482] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.294713 15223 raft_consensus.cc:504] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.295485 15223 leader_election.cc:288] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:09.296150 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0815524d0eb448438dae05f646f14580" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:09.296146 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0815524d0eb448438dae05f646f14580" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:09.296347 14599 raft_consensus.cc:2454] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:09.296386 14206 raft_consensus.cc:2454] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:09.296809 14272 leader_election.cc:302] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:09.297099 15223 raft_consensus.cc:2781] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.297196 15223 raft_consensus.cc:482] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.297288 15223 raft_consensus.cc:3037] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.299387 15223 raft_consensus.cc:504] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.299926 15223 leader_election.cc:288] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Requested vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:09.300361 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0815524d0eb448438dae05f646f14580" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:09.300410 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0815524d0eb448438dae05f646f14580" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:09.300565 14599 raft_consensus.cc:3037] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.300570 14206 raft_consensus.cc:3037] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.302402 14599 raft_consensus.cc:2454] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:09.302435 14206 raft_consensus.cc:2454] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:09.302837 14272 leader_election.cc:302] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:09.303128 15223 raft_consensus.cc:2781] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.303589 15223 raft_consensus.cc:686] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:09.303841 15223 consensus_queue.cc:237] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.306876 14058 catalog_manager.cc:5484] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:09.320240 15225 raft_consensus.cc:482] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.320459 15225 raft_consensus.cc:504] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.321286 15225 leader_election.cc:288] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:09.322391 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827a2fb0396441749a35977e2854bfb6" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:09.322613 14468 raft_consensus.cc:2454] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:09.323097 14141 leader_election.cc:302] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:09.323402 15225 raft_consensus.cc:2781] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.323547 15225 raft_consensus.cc:482] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.323666 15225 raft_consensus.cc:3037] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.326018 15225 raft_consensus.cc:504] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.326830 15225 leader_election.cc:288] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:09.327334 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827a2fb0396441749a35977e2854bfb6" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:09.327554 14468 raft_consensus.cc:3037] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.329583 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827a2fb0396441749a35977e2854bfb6" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:09.329753 14468 raft_consensus.cc:2454] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:09.329634 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827a2fb0396441749a35977e2854bfb6" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:09.329850 14599 raft_consensus.cc:2454] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:09.330302 14141 leader_election.cc:302] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:09.331079 15225 raft_consensus.cc:2781] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.331455 15225 raft_consensus.cc:686] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:09.331773 15225 consensus_queue.cc:237] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:09.334619 14058 catalog_manager.cc:5484] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:09.685797 15314 raft_consensus.cc:482] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:09.686013 15314 raft_consensus.cc:504] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.686942 15314 leader_election.cc:288] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:09.687593 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88890b0913c0462bade8208dfa0f64b7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:09.687866 14336 raft_consensus.cc:2454] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:09.688369 14535 leader_election.cc:302] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:09.688730 15314 raft_consensus.cc:2781] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:09.688841 15314 raft_consensus.cc:482] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:09.688941 15314 raft_consensus.cc:3037] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.691155 15314 raft_consensus.cc:504] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.691836 15314 leader_election.cc:288] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Requested vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:09.692286 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88890b0913c0462bade8208dfa0f64b7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:09.692482 14336 raft_consensus.cc:3037] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.693257 14206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88890b0913c0462bade8208dfa0f64b7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:09.693354 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88890b0913c0462bade8208dfa0f64b7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:09.693536 14206 raft_consensus.cc:2454] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:09.694711 14336 raft_consensus.cc:2454] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:09.695109 14535 leader_election.cc:302] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: b19bedd6f7f94bed9c727f6c7210cc8a
I20240502 08:18:09.695400 15314 raft_consensus.cc:2781] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:09.695746 15314 raft_consensus.cc:686] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:09.696017 15314 consensus_queue.cc:237] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:09.698774 14058 catalog_manager.cc:5484] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 0 to 1, leader changed from <none> to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:09.746201 14598 raft_consensus.cc:3037] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:09.748286 15225 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.749070 15225 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.755407 15223 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:09.763363 15223 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:10.134819 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:10.135138 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:10.142727 15415 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:10.152598 15314 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:10.157466 14205 raft_consensus.cc:3037] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.161326 15402 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
W20240502 08:18:10.473973 14270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 88989.4us

INSERT report
    rows total: 2000
    time total: 300.327 ms
  time per row: 0.150164 ms
I20240502 08:18:10.604820 14061 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:10.613850 14316 tablet_service.cc:1467] Processing CreateTablet for tablet b44cc3d7fc9944c7abf33b4da92b2f6f (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.613983 14449 tablet_service.cc:1467] Processing CreateTablet for tablet b44cc3d7fc9944c7abf33b4da92b2f6f (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.614352 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44cc3d7fc9944c7abf33b4da92b2f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.614420 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44cc3d7fc9944c7abf33b4da92b2f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.614869 14447 tablet_service.cc:1467] Processing CreateTablet for tablet 4f109afd932d4c77a6b7cd63c417bc70 (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.615300 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f109afd932d4c77a6b7cd63c417bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.615361 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 4f109afd932d4c77a6b7cd63c417bc70 (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.615770 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f109afd932d4c77a6b7cd63c417bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.618484 14185 tablet_service.cc:1467] Processing CreateTablet for tablet 4f109afd932d4c77a6b7cd63c417bc70 (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.618993 14185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f109afd932d4c77a6b7cd63c417bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.618943 14184 tablet_service.cc:1467] Processing CreateTablet for tablet b44cc3d7fc9944c7abf33b4da92b2f6f (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.619283 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44cc3d7fc9944c7abf33b4da92b2f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.624644 14317 tablet_service.cc:1467] Processing CreateTablet for tablet a635dc6d64724295be4b4a7f68d0fad6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.625070 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a635dc6d64724295be4b4a7f68d0fad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.626447 15478 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:10.626734 15477 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:10.627586 14579 tablet_service.cc:1467] Processing CreateTablet for tablet a635dc6d64724295be4b4a7f68d0fad6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.627986 14579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a635dc6d64724295be4b4a7f68d0fad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.628322 15478 tablet_bootstrap.cc:654] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.628353 15477 tablet_bootstrap.cc:654] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.630931 15478 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:10.631247 15478 ts_tablet_manager.cc:1367] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.005s	user 0.001s	sys 0.003s
I20240502 08:18:10.631951 14449 tablet_service.cc:1467] Processing CreateTablet for tablet a635dc6d64724295be4b4a7f68d0fad6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:10.632406 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a635dc6d64724295be4b4a7f68d0fad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:10.632555 15478 raft_consensus.cc:348] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.632923 15478 raft_consensus.cc:374] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.633093 15478 raft_consensus.cc:729] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.633498 15478 consensus_queue.cc:260] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.634403 15478 ts_tablet_manager.cc:1398] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:10.646275 15478 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:10.648460 15478 tablet_bootstrap.cc:654] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.655259 15482 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:10.656523 15481 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:10.657779 15478 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:10.657737 15477 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:10.657943 15477 ts_tablet_manager.cc:1367] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.031s	user 0.030s	sys 0.000s
I20240502 08:18:10.658015 15482 tablet_bootstrap.cc:654] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.658020 15478 ts_tablet_manager.cc:1367] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240502 08:18:10.658530 15481 tablet_bootstrap.cc:654] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.658901 15478 raft_consensus.cc:348] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.659250 15478 raft_consensus.cc:374] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.659389 15478 raft_consensus.cc:729] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.659759 15478 consensus_queue.cc:260] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.659758 15477 raft_consensus.cc:348] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.660101 15477 raft_consensus.cc:374] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.660238 15477 raft_consensus.cc:729] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.660646 15477 consensus_queue.cc:260] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.661366 15478 ts_tablet_manager.cc:1398] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:10.661541 15477 ts_tablet_manager.cc:1398] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:10.661901 15477 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:10.663815 15482 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:10.663969 15482 ts_tablet_manager.cc:1367] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.004s
I20240502 08:18:10.664935 15477 tablet_bootstrap.cc:654] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.664817 15482 raft_consensus.cc:348] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.665031 15482 raft_consensus.cc:374] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.665467 15481 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:10.665642 15481 ts_tablet_manager.cc:1367] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.009s	user 0.005s	sys 0.000s
I20240502 08:18:10.665948 15482 raft_consensus.cc:729] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.666289 15482 consensus_queue.cc:260] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.666539 15481 raft_consensus.cc:348] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.666796 15481 raft_consensus.cc:374] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.666888 15481 raft_consensus.cc:729] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.667173 15481 consensus_queue.cc:260] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.668599 15482 ts_tablet_manager.cc:1398] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:18:10.668933 15482 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:10.670439 15481 ts_tablet_manager.cc:1398] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:18:10.670794 15481 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:10.671504 15482 tablet_bootstrap.cc:654] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.671551 15477 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:10.671857 15477 ts_tablet_manager.cc:1367] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.010s	user 0.007s	sys 0.000s
I20240502 08:18:10.672605 15481 tablet_bootstrap.cc:654] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.672767 15477 raft_consensus.cc:348] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.673075 15477 raft_consensus.cc:374] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.673216 15477 raft_consensus.cc:729] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.673554 15477 consensus_queue.cc:260] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.674360 15477 ts_tablet_manager.cc:1398] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:10.675202 15477 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:10.676929 15477 tablet_bootstrap.cc:654] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:10.680320 15482 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:10.680506 15482 ts_tablet_manager.cc:1367] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.012s	user 0.004s	sys 0.004s
I20240502 08:18:10.681000 15481 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:10.681239 15481 ts_tablet_manager.cc:1367] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.006s
I20240502 08:18:10.681150 15482 raft_consensus.cc:348] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.681391 15482 raft_consensus.cc:374] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.681492 15482 raft_consensus.cc:729] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.681768 15482 consensus_queue.cc:260] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.682247 15477 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:10.682267 15482 ts_tablet_manager.cc:1398] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:10.682137 15481 raft_consensus.cc:348] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.682365 15481 raft_consensus.cc:374] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.682387 15477 ts_tablet_manager.cc:1367] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.008s	user 0.002s	sys 0.004s
I20240502 08:18:10.682440 15481 raft_consensus.cc:729] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.682708 15481 consensus_queue.cc:260] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.683151 15477 raft_consensus.cc:348] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.683352 15477 raft_consensus.cc:374] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:10.683449 15477 raft_consensus.cc:729] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:10.683704 15477 consensus_queue.cc:260] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.684212 15477 ts_tablet_manager.cc:1398] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:10.684404 15481 ts_tablet_manager.cc:1398] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:10.688905 15423 raft_consensus.cc:482] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.689034 15423 raft_consensus.cc:504] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.689605 15423 leader_election.cc:288] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:10.690094 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:10.690040 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:10.690306 14598 raft_consensus.cc:2454] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:10.690305 14205 raft_consensus.cc:2454] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:10.690776 14402 leader_election.cc:302] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:10.691062 15423 raft_consensus.cc:2781] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.691165 15423 raft_consensus.cc:482] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.691251 15423 raft_consensus.cc:3037] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.693164 15423 raft_consensus.cc:504] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.693706 15423 leader_election.cc:288] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Requested vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:10.694159 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:10.694164 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:10.694314 14205 raft_consensus.cc:3037] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.694334 14598 raft_consensus.cc:3037] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.696209 14205 raft_consensus.cc:2454] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:10.696215 14598 raft_consensus.cc:2454] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:10.696589 14400 leader_election.cc:302] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:10.696889 15423 raft_consensus.cc:2781] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.697590 15423 raft_consensus.cc:686] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:10.697899 15423 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:10.700676 14058 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67). New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:10.702200 15423 raft_consensus.cc:482] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.702363 15423 raft_consensus.cc:504] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.703032 15423 leader_election.cc:288] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:10.703439 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:10.703712 14335 raft_consensus.cc:2454] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:10.703720 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:10.703953 14205 raft_consensus.cc:2454] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:10.704198 14402 leader_election.cc:302] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde; no voters: 
I20240502 08:18:10.704460 15423 raft_consensus.cc:2781] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.704540 15423 raft_consensus.cc:482] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.704643 15423 raft_consensus.cc:3037] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.706454 15423 raft_consensus.cc:504] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.707047 15423 leader_election.cc:288] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:10.707470 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:10.707464 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:10.707664 14335 raft_consensus.cc:3037] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.707684 14205 raft_consensus.cc:3037] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.710529 14205 raft_consensus.cc:2454] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:10.710543 14335 raft_consensus.cc:2454] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:10.710963 14402 leader_election.cc:302] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde; no voters: 
I20240502 08:18:10.711242 15423 raft_consensus.cc:2781] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.711434 15423 raft_consensus.cc:686] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:10.711663 15423 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:10.714571 14060 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67). New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:10.882941 15423 raft_consensus.cc:482] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:10.883158 15423 raft_consensus.cc:504] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.883852 15423 leader_election.cc:288] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:10.884464 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:10.884582 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:10.884735 14598 raft_consensus.cc:2454] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:10.884835 14335 raft_consensus.cc:2454] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:10.885180 14402 leader_election.cc:302] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:10.885452 15423 raft_consensus.cc:2781] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:10.885576 15423 raft_consensus.cc:482] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:10.885665 15423 raft_consensus.cc:3037] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.887677 15423 raft_consensus.cc:504] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.888197 15423 leader_election.cc:288] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Requested vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:10.888638 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:10.888686 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:10.888825 14598 raft_consensus.cc:3037] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.888900 14335 raft_consensus.cc:3037] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:10.890944 14598 raft_consensus.cc:2454] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:10.891060 14335 raft_consensus.cc:2454] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:10.891429 14402 leader_election.cc:302] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:10.891816 15423 raft_consensus.cc:2781] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:10.891968 15423 raft_consensus.cc:686] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:10.892278 15423 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:10.895023 14060 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67). New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:11.093815 15423 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.098846 15423 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.209548 15423 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.214381 15423 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:11.287690 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:11.288028 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:11.295538 15494 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:11.326301 15490 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:11.332109 15490 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
W20240502 08:18:11.545384 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.66:44269, user_credentials={real_user=slave}} blocked reactor thread for 78589.7us

INSERT report
    rows total: 2000
    time total: 226.025 ms
  time per row: 0.113012 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:18:12.656616 14063 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_4"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:12.666451 14185 tablet_service.cc:1467] Processing CreateTablet for tablet fec3033140544f4994057d5b22973fff (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.667017 14185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec3033140544f4994057d5b22973fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.667029 14447 tablet_service.cc:1467] Processing CreateTablet for tablet fec3033140544f4994057d5b22973fff (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.667383 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec3033140544f4994057d5b22973fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.667300 14317 tablet_service.cc:1467] Processing CreateTablet for tablet fec3033140544f4994057d5b22973fff (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.667721 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec3033140544f4994057d5b22973fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.671196 14184 tablet_service.cc:1467] Processing CreateTablet for tablet e6d3c011ad9546e3b69e23cbef587dbc (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.671480 14316 tablet_service.cc:1467] Processing CreateTablet for tablet e6d3c011ad9546e3b69e23cbef587dbc (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.671597 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d3c011ad9546e3b69e23cbef587dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.671819 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d3c011ad9546e3b69e23cbef587dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.672595 14186 tablet_service.cc:1467] Processing CreateTablet for tablet 6379601e2b2d4bdabde2773fcbe06fb4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.672956 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6379601e2b2d4bdabde2773fcbe06fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.679744 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 6379601e2b2d4bdabde2773fcbe06fb4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.679808 14578 tablet_service.cc:1467] Processing CreateTablet for tablet e6d3c011ad9546e3b69e23cbef587dbc (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.679808 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 6379601e2b2d4bdabde2773fcbe06fb4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:12.680204 14579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6379601e2b2d4bdabde2773fcbe06fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.680220 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6379601e2b2d4bdabde2773fcbe06fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.680437 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d3c011ad9546e3b69e23cbef587dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:12.685917 15562 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:12.688261 15562 tablet_bootstrap.cc:654] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.689965 15565 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:12.690325 15563 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:12.691854 15565 tablet_bootstrap.cc:654] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.692476 15563 tablet_bootstrap.cc:654] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.697211 15564 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:12.700523 15564 tablet_bootstrap.cc:654] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.700729 15563 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:12.700953 15563 ts_tablet_manager.cc:1367] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240502 08:18:12.701874 15565 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:12.701817 15563 raft_consensus.cc:348] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.702032 15563 raft_consensus.cc:374] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.702129 15563 raft_consensus.cc:729] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.702296 15565 ts_tablet_manager.cc:1367] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240502 08:18:12.702365 15563 consensus_queue.cc:260] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.702962 15563 ts_tablet_manager.cc:1398] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:12.703016 15562 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:12.703285 15562 ts_tablet_manager.cc:1367] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.003s
I20240502 08:18:12.703308 15563 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:12.704420 15562 raft_consensus.cc:348] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.704790 15562 raft_consensus.cc:374] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.704905 15563 tablet_bootstrap.cc:654] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.704964 15562 raft_consensus.cc:729] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.705317 15562 consensus_queue.cc:260] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.706147 15562 ts_tablet_manager.cc:1398] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240502 08:18:12.706584 15562 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:12.708014 15565 raft_consensus.cc:348] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.708314 15565 raft_consensus.cc:374] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.708436 15565 raft_consensus.cc:729] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.708775 15565 consensus_queue.cc:260] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.709604 15562 tablet_bootstrap.cc:654] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.714130 15565 ts_tablet_manager.cc:1398] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.001s
I20240502 08:18:12.714789 15562 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:12.715030 15562 ts_tablet_manager.cc:1367] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.009s	user 0.001s	sys 0.005s
I20240502 08:18:12.715260 15565 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:12.715605 15563 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:12.715806 15563 ts_tablet_manager.cc:1367] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.013s	user 0.002s	sys 0.009s
I20240502 08:18:12.716523 15562 raft_consensus.cc:348] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.716655 15563 raft_consensus.cc:348] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.716812 15562 raft_consensus.cc:374] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.716902 15563 raft_consensus.cc:374] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.716980 15563 raft_consensus.cc:729] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.717020 15562 raft_consensus.cc:729] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.717319 15563 consensus_queue.cc:260] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.717373 15562 consensus_queue.cc:260] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.717944 15563 ts_tablet_manager.cc:1398] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:18:12.718122 15562 ts_tablet_manager.cc:1398] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:12.718225 15565 tablet_bootstrap.cc:654] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.718250 15563 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:12.719269 15564 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:12.719471 15564 ts_tablet_manager.cc:1367] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.001s
I20240502 08:18:12.719841 15563 tablet_bootstrap.cc:654] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.720490 15564 raft_consensus.cc:348] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.720757 15564 raft_consensus.cc:374] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.720870 15564 raft_consensus.cc:729] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.721144 15564 consensus_queue.cc:260] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.721868 15564 ts_tablet_manager.cc:1398] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:12.722227 15564 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:12.724676 15564 tablet_bootstrap.cc:654] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:12.730047 15564 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:12.730372 15564 ts_tablet_manager.cc:1367] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.004s
I20240502 08:18:12.732080 15564 raft_consensus.cc:348] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.732515 15564 raft_consensus.cc:374] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.732683 15564 raft_consensus.cc:729] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.733098 15564 consensus_queue.cc:260] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.734082 15564 ts_tablet_manager.cc:1398] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 08:18:12.743638 15565 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:12.743741 15563 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:12.743839 15565 ts_tablet_manager.cc:1367] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.029s	user 0.010s	sys 0.014s
I20240502 08:18:12.743909 15563 ts_tablet_manager.cc:1367] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.026s	user 0.010s	sys 0.014s
I20240502 08:18:12.744585 15565 raft_consensus.cc:348] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.744709 15563 raft_consensus.cc:348] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.744854 15565 raft_consensus.cc:374] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.744933 15565 raft_consensus.cc:729] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.744936 15563 raft_consensus.cc:374] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:12.745034 15563 raft_consensus.cc:729] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:12.745191 15565 consensus_queue.cc:260] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.745260 15563 consensus_queue.cc:260] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.745772 15565 ts_tablet_manager.cc:1398] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:12.746407 15563 ts_tablet_manager.cc:1398] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:18:12.762385 15573 raft_consensus.cc:482] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.762696 15573 raft_consensus.cc:504] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.763403 15573 leader_election.cc:288] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:12.763969 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3033140544f4994057d5b22973fff" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:12.764010 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3033140544f4994057d5b22973fff" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:12.764185 14205 raft_consensus.cc:2454] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:12.764246 14335 raft_consensus.cc:2454] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:12.764681 14400 leader_election.cc:302] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:12.765012 15573 raft_consensus.cc:2781] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.765128 15573 raft_consensus.cc:482] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.765242 15573 raft_consensus.cc:3037] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.767385 15573 raft_consensus.cc:504] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.767889 15573 leader_election.cc:288] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Requested vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:12.768369 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3033140544f4994057d5b22973fff" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:12.768373 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fec3033140544f4994057d5b22973fff" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:12.768549 14335 raft_consensus.cc:3037] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.768566 14205 raft_consensus.cc:3037] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.770435 14205 raft_consensus.cc:2454] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:12.770437 14335 raft_consensus.cc:2454] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:12.770880 14402 leader_election.cc:302] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde; no voters: 
I20240502 08:18:12.771273 15573 raft_consensus.cc:2781] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.771409 15573 raft_consensus.cc:686] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:12.771684 15573 consensus_queue.cc:237] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:12.774899 14063 catalog_manager.cc:5484] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67). New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:12.790474 15314 raft_consensus.cc:482] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.790692 15314 raft_consensus.cc:504] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.791327 15314 leader_election.cc:288] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:12.791846 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d3c011ad9546e3b69e23cbef587dbc" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:12.791846 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d3c011ad9546e3b69e23cbef587dbc" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:12.792115 14335 raft_consensus.cc:2454] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:12.792120 14205 raft_consensus.cc:2454] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:12.792557 14533 leader_election.cc:302] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b19bedd6f7f94bed9c727f6c7210cc8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:12.792836 15314 raft_consensus.cc:2781] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.792949 15314 raft_consensus.cc:482] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.793032 15314 raft_consensus.cc:3037] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.794919 15314 raft_consensus.cc:504] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.795464 15314 leader_election.cc:288] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:12.795933 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d3c011ad9546e3b69e23cbef587dbc" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:12.796011 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d3c011ad9546e3b69e23cbef587dbc" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:12.796146 14335 raft_consensus.cc:3037] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.796248 14205 raft_consensus.cc:3037] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.798224 14335 raft_consensus.cc:2454] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:12.798728 14535 leader_election.cc:302] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:12.799078 15314 raft_consensus.cc:2781] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.799283 14205 raft_consensus.cc:2454] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:12.799611 15314 raft_consensus.cc:686] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:12.799928 15314 consensus_queue.cc:237] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:12.802685 14063 catalog_manager.cc:5484] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 0 to 1, leader changed from <none> to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:12.823113 15453 raft_consensus.cc:482] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:12.823310 15453 raft_consensus.cc:504] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.823959 15453 leader_election.cc:288] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:12.824515 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6379601e2b2d4bdabde2773fcbe06fb4" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:12.824551 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6379601e2b2d4bdabde2773fcbe06fb4" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:12.824755 14598 raft_consensus.cc:2454] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:12.824791 14468 raft_consensus.cc:2454] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:12.825218 14141 leader_election.cc:302] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b19bedd6f7f94bed9c727f6c7210cc8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:12.825506 15453 raft_consensus.cc:2781] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:12.825615 15453 raft_consensus.cc:482] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:12.825707 15453 raft_consensus.cc:3037] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.827683 15453 raft_consensus.cc:504] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.828233 15453 leader_election.cc:288] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:12.828771 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6379601e2b2d4bdabde2773fcbe06fb4" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:12.828828 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6379601e2b2d4bdabde2773fcbe06fb4" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:12.828994 14468 raft_consensus.cc:3037] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.828998 14598 raft_consensus.cc:3037] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:12.831013 14598 raft_consensus.cc:2454] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:12.831175 14468 raft_consensus.cc:2454] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:12.831547 14141 leader_election.cc:302] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b19bedd6f7f94bed9c727f6c7210cc8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:12.831852 15453 raft_consensus.cc:2781] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:12.832250 15453 raft_consensus.cc:686] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:12.832573 15453 consensus_queue.cc:237] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:12.835328 14063 catalog_manager.cc:5484] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:13.163724 15573 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.172693 15573 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:13.203435 15581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:13.203804 15581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:13.211563 15581 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 08:18:13.292737 15453 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.298272 15453 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.369520 15314 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:13.445669 14335 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:18:13.447857 15629 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 304.261 ms
  time per row: 0.15213 ms
I20240502 08:18:13.665386 14061 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_5"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:13.674224 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 97d7768da26e4fff966e494446a278f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.674386 14447 tablet_service.cc:1467] Processing CreateTablet for tablet 97d7768da26e4fff966e494446a278f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.674762 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d7768da26e4fff966e494446a278f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.674857 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d7768da26e4fff966e494446a278f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.675053 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 810c5f8a81ba4e0ab06129c0178f12e7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.675401 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810c5f8a81ba4e0ab06129c0178f12e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.676086 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 97d7768da26e4fff966e494446a278f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.676519 14579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d7768da26e4fff966e494446a278f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.682112 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 810c5f8a81ba4e0ab06129c0178f12e7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.682250 14315 tablet_service.cc:1467] Processing CreateTablet for tablet 0b15694b0b284ecebd899522dc69d422 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.682549 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810c5f8a81ba4e0ab06129c0178f12e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.682722 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b15694b0b284ecebd899522dc69d422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.682730 15651 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:13.683919 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 0b15694b0b284ecebd899522dc69d422 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.684351 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b15694b0b284ecebd899522dc69d422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.685848 14316 tablet_service.cc:1467] Processing CreateTablet for tablet 810c5f8a81ba4e0ab06129c0178f12e7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.686201 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810c5f8a81ba4e0ab06129c0178f12e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.686851 15651 tablet_bootstrap.cc:654] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.691155 14448 tablet_service.cc:1467] Processing CreateTablet for tablet 0b15694b0b284ecebd899522dc69d422 (DEFAULT_TABLE table=rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:13.691726 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b15694b0b284ecebd899522dc69d422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:13.694068 15652 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:13.696449 15655 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:13.697014 15654 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:13.697458 15652 tablet_bootstrap.cc:654] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.699378 15654 tablet_bootstrap.cc:654] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.701462 15655 tablet_bootstrap.cc:654] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.711874 15652 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:13.712220 15652 ts_tablet_manager.cc:1367] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.012s
I20240502 08:18:13.713510 15652 raft_consensus.cc:348] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.713992 15652 raft_consensus.cc:374] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.714162 15652 raft_consensus.cc:729] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.714584 15652 consensus_queue.cc:260] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.715544 15652 ts_tablet_manager.cc:1398] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:13.716074 15652 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:13.717183 15655 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:13.717437 15655 ts_tablet_manager.cc:1367] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.005s
I20240502 08:18:13.718420 15652 tablet_bootstrap.cc:654] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.718472 15655 raft_consensus.cc:348] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.718847 15655 raft_consensus.cc:374] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.719003 15655 raft_consensus.cc:729] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.719300 15655 consensus_queue.cc:260] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.720217 15655 ts_tablet_manager.cc:1398] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:13.724146 15651 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:13.724349 15651 ts_tablet_manager.cc:1367] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.042s	user 0.026s	sys 0.007s
I20240502 08:18:13.726529 15654 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:13.726536 15651 raft_consensus.cc:348] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.726722 15654 ts_tablet_manager.cc:1367] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.004s
I20240502 08:18:13.726814 15651 raft_consensus.cc:374] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.726922 15651 raft_consensus.cc:729] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.727174 15651 consensus_queue.cc:260] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.727537 15654 raft_consensus.cc:348] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.727754 15654 raft_consensus.cc:374] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.727782 15651 ts_tablet_manager.cc:1398] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:13.727914 15654 raft_consensus.cc:729] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.728137 15651 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:13.728207 15654 consensus_queue.cc:260] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.728814 15654 ts_tablet_manager.cc:1398] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:13.729192 15654 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:13.729898 15651 tablet_bootstrap.cc:654] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.731036 15654 tablet_bootstrap.cc:654] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.735901 15652 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:13.736553 15652 ts_tablet_manager.cc:1367] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.021s	user 0.002s	sys 0.006s
I20240502 08:18:13.737740 15652 raft_consensus.cc:348] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.738031 15652 raft_consensus.cc:374] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.738176 15654 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:13.738168 15652 raft_consensus.cc:729] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.738287 15651 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:13.738350 15654 ts_tablet_manager.cc:1367] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.009s	user 0.004s	sys 0.003s
I20240502 08:18:13.738477 15651 ts_tablet_manager.cc:1367] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.004s
I20240502 08:18:13.738536 15652 consensus_queue.cc:260] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.739189 15654 raft_consensus.cc:348] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.739313 15652 ts_tablet_manager.cc:1398] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:13.739279 15651 raft_consensus.cc:348] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.739514 15654 raft_consensus.cc:374] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.739550 15651 raft_consensus.cc:374] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.739658 15652 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:13.739662 15654 raft_consensus.cc:729] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.739851 15651 raft_consensus.cc:729] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.740110 15651 consensus_queue.cc:260] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.740537 15651 ts_tablet_manager.cc:1398] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:18:13.740804 15651 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:13.742259 15652 tablet_bootstrap.cc:654] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.742286 15651 tablet_bootstrap.cc:654] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:13.744139 15651 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:13.744336 15651 ts_tablet_manager.cc:1367] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 08:18:13.744720 15654 consensus_queue.cc:260] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.745122 15651 raft_consensus.cc:348] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.745362 15651 raft_consensus.cc:374] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.745476 15651 raft_consensus.cc:729] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.745716 15651 consensus_queue.cc:260] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.746171 15651 ts_tablet_manager.cc:1398] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:18:13.747238 15654 ts_tablet_manager.cc:1398] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 08:18:13.749183 15652 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:13.749938 15652 ts_tablet_manager.cc:1367] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.006s
I20240502 08:18:13.750602 15652 raft_consensus.cc:348] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.750808 15652 raft_consensus.cc:374] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:13.750906 15652 raft_consensus.cc:729] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:13.751152 15652 consensus_queue.cc:260] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.751657 15652 ts_tablet_manager.cc:1398] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:18:13.754240 15643 raft_consensus.cc:482] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.754431 15643 raft_consensus.cc:504] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.755362 15643 leader_election.cc:288] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:13.755846 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b15694b0b284ecebd899522dc69d422" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:13.755841 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b15694b0b284ecebd899522dc69d422" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:13.756104 14469 raft_consensus.cc:2454] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:13.756115 14335 raft_consensus.cc:2454] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:13.756554 14141 leader_election.cc:302] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:13.756841 15643 raft_consensus.cc:2781] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.756960 15643 raft_consensus.cc:482] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.757050 15643 raft_consensus.cc:3037] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.759145 15643 raft_consensus.cc:504] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.759707 15643 leader_election.cc:288] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:13.760135 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b15694b0b284ecebd899522dc69d422" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:13.760344 14335 raft_consensus.cc:3037] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.762343 14335 raft_consensus.cc:2454] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:13.762712 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b15694b0b284ecebd899522dc69d422" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:13.762784 14141 leader_election.cc:302] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:13.762960 14469 raft_consensus.cc:3037] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.763108 15643 raft_consensus.cc:2781] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.763275 15643 raft_consensus.cc:686] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:13.763583 15643 consensus_queue.cc:237] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:13.765588 14469 raft_consensus.cc:2454] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:13.766902 14063 catalog_manager.cc:5484] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:13.813583 15629 raft_consensus.cc:482] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.813808 15629 raft_consensus.cc:504] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.814539 15629 leader_election.cc:288] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:13.815106 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810c5f8a81ba4e0ab06129c0178f12e7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:13.815351 14335 raft_consensus.cc:2454] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:13.815297 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810c5f8a81ba4e0ab06129c0178f12e7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:13.815531 14469 raft_consensus.cc:2454] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:13.815815 14535 leader_election.cc:302] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:13.816128 15629 raft_consensus.cc:2781] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.816231 15629 raft_consensus.cc:482] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.816324 15629 raft_consensus.cc:3037] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.818384 15629 raft_consensus.cc:504] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.819130 15629 leader_election.cc:288] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:13.819577 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810c5f8a81ba4e0ab06129c0178f12e7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:13.819617 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810c5f8a81ba4e0ab06129c0178f12e7" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:13.819779 14335 raft_consensus.cc:3037] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.819856 14469 raft_consensus.cc:3037] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.821734 14335 raft_consensus.cc:2454] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:13.821861 14469 raft_consensus.cc:2454] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:13.822146 14535 leader_election.cc:302] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:13.822433 15629 raft_consensus.cc:2781] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.822597 15629 raft_consensus.cc:686] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:13.822899 15629 consensus_queue.cc:237] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.825984 14061 catalog_manager.cc:5484] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 0 to 1, leader changed from <none> to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:13.848332 15452 raft_consensus.cc:482] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:13.848521 15452 raft_consensus.cc:504] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.849329 15452 leader_election.cc:288] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:13.850697 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d7768da26e4fff966e494446a278f2" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:13.850963 14598 raft_consensus.cc:2454] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:13.851433 14272 leader_election.cc:302] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:13.851724 15452 raft_consensus.cc:2781] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:13.851842 15452 raft_consensus.cc:482] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:13.851943 15452 raft_consensus.cc:3037] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.854491 15452 raft_consensus.cc:504] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.855170 15452 leader_election.cc:288] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:13.855710 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d7768da26e4fff966e494446a278f2" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:13.855942 14598 raft_consensus.cc:3037] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.858373 14598 raft_consensus.cc:2454] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:13.858767 14272 leader_election.cc:302] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:13.859021 15667 raft_consensus.cc:2781] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:13.859195 15667 raft_consensus.cc:686] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:13.859544 15667 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:13.862398 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d7768da26e4fff966e494446a278f2" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:13.862335 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97d7768da26e4fff966e494446a278f2" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:13.862622 14468 raft_consensus.cc:3037] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:13.865492 14468 raft_consensus.cc:2454] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:13.871799 14061 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.147575 15643 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.164451 15643 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.265990 15629 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:14.279058 15669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:14.279515 15669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:18:14.282037 15629 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:14.289774 15669 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 08:18:14.372597 15667 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:14.406443 15704 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 245.065 ms
  time per row: 0.122532 ms
I20240502 08:18:14.695586 14063 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_6"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:14.704066 14184 tablet_service.cc:1467] Processing CreateTablet for tablet cf7124419bae40dfaaabdd2e673a3eb0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.704581 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7124419bae40dfaaabdd2e673a3eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.704985 14317 tablet_service.cc:1467] Processing CreateTablet for tablet cf7124419bae40dfaaabdd2e673a3eb0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.705251 14448 tablet_service.cc:1467] Processing CreateTablet for tablet 7a92e7e903594d84a59f3ed058a5a3ec (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.705257 14578 tablet_service.cc:1467] Processing CreateTablet for tablet cf7124419bae40dfaaabdd2e673a3eb0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.705405 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7124419bae40dfaaabdd2e673a3eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.705657 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7124419bae40dfaaabdd2e673a3eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.705669 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a92e7e903594d84a59f3ed058a5a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.711668 14579 tablet_service.cc:1467] Processing CreateTablet for tablet 7a92e7e903594d84a59f3ed058a5a3ec (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.712204 14579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a92e7e903594d84a59f3ed058a5a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.713179 14315 tablet_service.cc:1467] Processing CreateTablet for tablet 5341f4bf3c9643859859ed0d9887cfca (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.713557 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5341f4bf3c9643859859ed0d9887cfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.714126 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 5341f4bf3c9643859859ed0d9887cfca (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.714550 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5341f4bf3c9643859859ed0d9887cfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.716919 14186 tablet_service.cc:1467] Processing CreateTablet for tablet 7a92e7e903594d84a59f3ed058a5a3ec (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.717324 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a92e7e903594d84a59f3ed058a5a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.721120 15738 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:14.721992 15735 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:14.723946 15735 tablet_bootstrap.cc:654] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.724059 15738 tablet_bootstrap.cc:654] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.725838 15736 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:14.725972 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 5341f4bf3c9643859859ed0d9887cfca (DEFAULT_TABLE table=rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:14.726696 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5341f4bf3c9643859859ed0d9887cfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:14.728494 15736 tablet_bootstrap.cc:654] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.733945 15737 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:14.733966 15738 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:14.734133 15738 ts_tablet_manager.cc:1367] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.013s	user 0.006s	sys 0.000s
I20240502 08:18:14.734386 15735 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:14.734534 15735 ts_tablet_manager.cc:1367] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.000s
I20240502 08:18:14.735154 15738 raft_consensus.cc:348] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.735379 15735 raft_consensus.cc:348] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.735553 15736 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:14.735517 15738 raft_consensus.cc:374] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.735699 15735 raft_consensus.cc:374] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.735733 15736 ts_tablet_manager.cc:1367] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.003s
I20240502 08:18:14.735724 15738 raft_consensus.cc:729] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.735895 15735 raft_consensus.cc:729] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.736107 15737 tablet_bootstrap.cc:654] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.736164 15738 consensus_queue.cc:260] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.736239 15735 consensus_queue.cc:260] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.736963 15735 ts_tablet_manager.cc:1398] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:14.737751 15735 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:14.736507 15736 raft_consensus.cc:348] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.738162 15736 raft_consensus.cc:374] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.738278 15736 raft_consensus.cc:729] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.738622 15736 consensus_queue.cc:260] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.739341 15736 ts_tablet_manager.cc:1398] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:18:14.739686 15736 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:14.740253 15735 tablet_bootstrap.cc:654] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.741381 15736 tablet_bootstrap.cc:654] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.745613 15735 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:14.745846 15735 ts_tablet_manager.cc:1367] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.003s
I20240502 08:18:14.746634 15735 raft_consensus.cc:348] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.746996 15735 raft_consensus.cc:374] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.747118 15735 raft_consensus.cc:729] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.747511 15735 consensus_queue.cc:260] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.748417 15735 ts_tablet_manager.cc:1398] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:14.748875 15735 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:14.751066 15735 tablet_bootstrap.cc:654] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.753353 15737 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:14.753625 15737 ts_tablet_manager.cc:1367] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.007s
I20240502 08:18:14.754024 15738 ts_tablet_manager.cc:1398] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.020s	user 0.002s	sys 0.000s
I20240502 08:18:14.755034 15737 raft_consensus.cc:348] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.755427 15736 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:14.755450 15737 raft_consensus.cc:374] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.755662 15736 ts_tablet_manager.cc:1367] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.016s	user 0.001s	sys 0.012s
I20240502 08:18:14.755661 15737 raft_consensus.cc:729] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.756043 15737 consensus_queue.cc:260] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.756670 15453 raft_consensus.cc:482] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.756639 15736 raft_consensus.cc:348] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.756929 15736 raft_consensus.cc:374] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.756891 15453 raft_consensus.cc:504] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.757033 15736 raft_consensus.cc:729] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.757329 15736 consensus_queue.cc:260] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.757623 15453 leader_election.cc:288] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:14.758415 15737 ts_tablet_manager.cc:1398] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:18:14.758471 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a92e7e903594d84a59f3ed058a5a3ec" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:14.758764 15736 ts_tablet_manager.cc:1398] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:18:14.758949 15737 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
W20240502 08:18:14.759420 14141 leader_election.cc:341] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:14.759820 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a92e7e903594d84a59f3ed058a5a3ec" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:14.760099 14468 raft_consensus.cc:2454] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:14.760720 14141 leader_election.cc:302] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: e79a5862de19490b89a9ea97860d776b
I20240502 08:18:14.761065 15453 raft_consensus.cc:2781] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.761166 15453 raft_consensus.cc:482] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.761298 15453 raft_consensus.cc:3037] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.763712 15735 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:14.763926 15735 ts_tablet_manager.cc:1367] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.000s
I20240502 08:18:14.764036 15453 raft_consensus.cc:504] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.764734 15453 leader_election.cc:288] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:14.764981 15735 raft_consensus.cc:348] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.765082 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a92e7e903594d84a59f3ed058a5a3ec" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:14.765159 15737 tablet_bootstrap.cc:654] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.765198 15735 raft_consensus.cc:374] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.765297 15735 raft_consensus.cc:729] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.765527 15735 consensus_queue.cc:260] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.766124 15735 ts_tablet_manager.cc:1398] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:18:14.767256 15737 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:14.767408 15737 ts_tablet_manager.cc:1367] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.001s
W20240502 08:18:14.767798 14141 leader_election.cc:341] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:14.767885 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a92e7e903594d84a59f3ed058a5a3ec" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:14.768078 14468 raft_consensus.cc:3037] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.768189 15737 raft_consensus.cc:348] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.768393 15737 raft_consensus.cc:374] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.768464 15737 raft_consensus.cc:729] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.768724 15737 consensus_queue.cc:260] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.769847 15737 ts_tablet_manager.cc:1398] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:18:14.770164 15737 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:14.770318 14468 raft_consensus.cc:2454] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:14.770736 14141 leader_election.cc:302] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: e79a5862de19490b89a9ea97860d776b
I20240502 08:18:14.771772 15453 raft_consensus.cc:2781] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.771955 15453 raft_consensus.cc:686] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:14.771884 15737 tablet_bootstrap.cc:654] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:14.772222 15453 consensus_queue.cc:237] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.775034 15737 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:14.775188 15737 ts_tablet_manager.cc:1367] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:18:14.775892 15737 raft_consensus.cc:348] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.776109 15737 raft_consensus.cc:374] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:14.776201 15737 raft_consensus.cc:729] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:14.776425 15737 consensus_queue.cc:260] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.776582 15667 raft_consensus.cc:482] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.776803 15667 raft_consensus.cc:504] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.777062 15737 ts_tablet_manager.cc:1398] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:18:14.777529 15667 leader_election.cc:288] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:14.778141 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5341f4bf3c9643859859ed0d9887cfca" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:14.778323 14205 raft_consensus.cc:2454] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:14.778292 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5341f4bf3c9643859859ed0d9887cfca" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:14.778540 14598 raft_consensus.cc:2454] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:14.778793 14270 leader_election.cc:302] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:14.779213 15667 raft_consensus.cc:2781] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.779332 15667 raft_consensus.cc:482] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.779431 15667 raft_consensus.cc:3037] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.779047 14062 catalog_manager.cc:5484] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.781564 15667 raft_consensus.cc:504] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.782735 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5341f4bf3c9643859859ed0d9887cfca" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:14.782938 14598 raft_consensus.cc:3037] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.785219 14598 raft_consensus.cc:2454] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:14.785686 14272 leader_election.cc:302] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:14.785768 15667 leader_election.cc:288] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Requested vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:14.786057 15703 raft_consensus.cc:2781] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.786197 15703 raft_consensus.cc:686] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:14.786482 15703 consensus_queue.cc:237] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:14.788012 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5341f4bf3c9643859859ed0d9887cfca" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:14.788249 14205 raft_consensus.cc:3037] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.790592 14205 raft_consensus.cc:2454] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:14.795135 14061 catalog_manager.cc:5484] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:14.836423 15703 raft_consensus.cc:482] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.836647 15703 raft_consensus.cc:504] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.837330 15703 leader_election.cc:288] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:14.837889 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7124419bae40dfaaabdd2e673a3eb0" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:14.837931 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7124419bae40dfaaabdd2e673a3eb0" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:14.838137 14205 raft_consensus.cc:2454] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:14.838148 14598 raft_consensus.cc:2454] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:14.838546 14272 leader_election.cc:302] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:14.838894 15703 raft_consensus.cc:2781] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:14.838976 15703 raft_consensus.cc:482] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:14.839079 15703 raft_consensus.cc:3037] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.840965 15703 raft_consensus.cc:504] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.841558 15703 leader_election.cc:288] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Requested vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:14.841989 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7124419bae40dfaaabdd2e673a3eb0" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:14.842077 14598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7124419bae40dfaaabdd2e673a3eb0" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:14.842224 14205 raft_consensus.cc:3037] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.842311 14598 raft_consensus.cc:3037] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.844336 14205 raft_consensus.cc:2454] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:14.844779 14270 leader_election.cc:302] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:14.845108 15703 raft_consensus.cc:2781] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:14.845115 14598 raft_consensus.cc:2454] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:14.845474 15703 raft_consensus.cc:686] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:14.845803 15703 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:14.849371 14061 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:14.975989 15681 raft_consensus.cc:482] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:14.976189 15681 raft_consensus.cc:504] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:14.976889 15681 leader_election.cc:288] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:14.977316 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a92e7e903594d84a59f3ed058a5a3ec" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:14.977469 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a92e7e903594d84a59f3ed058a5a3ec" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:14.977803 14468 raft_consensus.cc:2379] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e79a5862de19490b89a9ea97860d776b in current term 1: Already voted for candidate b19bedd6f7f94bed9c727f6c7210cc8a in this term.
I20240502 08:18:14.978261 14535 leader_election.cc:302] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e79a5862de19490b89a9ea97860d776b; no voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a
I20240502 08:18:14.978592 15681 raft_consensus.cc:3037] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:14.980652 15681 raft_consensus.cc:2726] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:15.194542 15453 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.201254 15453 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.212495 15703 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.224695 15667 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:15.243758 15667 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:15.251896 15751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:15.252221 15751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:18:15.260862 15703 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:15.263011 15751 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
W20240502 08:18:15.539332 14141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 88038.8us

INSERT report
    rows total: 2000
    time total: 286.934 ms
  time per row: 0.143467 ms
I20240502 08:18:15.700795 14063 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_7"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:15.709383 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1fecce883346459e21c29b98d95b7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.709635 14448 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1fecce883346459e21c29b98d95b7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.709888 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1fecce883346459e21c29b98d95b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.710041 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1fecce883346459e21c29b98d95b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.710960 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 1017d794c61345959c52f062dd39ef35 (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.711267 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 84eaabbe91524d129a170ad3519c265a (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.711395 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017d794c61345959c52f062dd39ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.711614 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84eaabbe91524d129a170ad3519c265a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.714352 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1fecce883346459e21c29b98d95b7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.714818 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1fecce883346459e21c29b98d95b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.715478 14449 tablet_service.cc:1467] Processing CreateTablet for tablet 1017d794c61345959c52f062dd39ef35 (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.716372 14447 tablet_service.cc:1467] Processing CreateTablet for tablet 84eaabbe91524d129a170ad3519c265a (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.716765 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017d794c61345959c52f062dd39ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.717708 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84eaabbe91524d129a170ad3519c265a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.718058 14315 tablet_service.cc:1467] Processing CreateTablet for tablet 84eaabbe91524d129a170ad3519c265a (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.718546 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84eaabbe91524d129a170ad3519c265a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.722095 14186 tablet_service.cc:1467] Processing CreateTablet for tablet 1017d794c61345959c52f062dd39ef35 (DEFAULT_TABLE table=rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:15.722633 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017d794c61345959c52f062dd39ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:15.742388 15814 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:15.744616 15814 tablet_bootstrap.cc:654] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.753327 15814 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:15.753515 15814 ts_tablet_manager.cc:1367] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.003s
I20240502 08:18:15.754688 15816 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:15.755016 15814 raft_consensus.cc:348] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.755262 15814 raft_consensus.cc:374] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.755380 15814 raft_consensus.cc:729] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.755625 15814 consensus_queue.cc:260] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.756207 15814 ts_tablet_manager.cc:1398] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:15.756565 15814 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:15.758033 15816 tablet_bootstrap.cc:654] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.758111 15815 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:15.758407 15814 tablet_bootstrap.cc:654] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.760190 15815 tablet_bootstrap.cc:654] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.765717 15815 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:15.765893 15815 ts_tablet_manager.cc:1367] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.008s	user 0.003s	sys 0.000s
I20240502 08:18:15.766821 15815 raft_consensus.cc:348] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:15.767148 15815 raft_consensus.cc:374] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.767321 15815 raft_consensus.cc:729] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.767642 15815 consensus_queue.cc:260] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:15.768357 15815 ts_tablet_manager.cc:1398] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 08:18:15.768707 15815 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:15.768744 15818 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:15.768774 15814 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:15.768959 15814 ts_tablet_manager.cc:1367] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240502 08:18:15.769810 15814 raft_consensus.cc:348] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:15.770062 15814 raft_consensus.cc:374] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.770150 15814 raft_consensus.cc:729] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.770504 15815 tablet_bootstrap.cc:654] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.770442 15814 consensus_queue.cc:260] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:15.771060 15814 ts_tablet_manager.cc:1398] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:15.771477 15814 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:15.772607 15818 tablet_bootstrap.cc:654] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.773097 15814 tablet_bootstrap.cc:654] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.775445 15816 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:15.775652 15816 ts_tablet_manager.cc:1367] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.021s	user 0.002s	sys 0.008s
I20240502 08:18:15.777289 15573 raft_consensus.cc:482] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.777493 15573 raft_consensus.cc:504] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.778383 15573 leader_election.cc:288] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:15.779004 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1fecce883346459e21c29b98d95b7b" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:15.779047 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1fecce883346459e21c29b98d95b7b" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:15.779691 15814 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
W20240502 08:18:15.779886 14402 leader_election.cc:341] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:15.779892 15814 ts_tablet_manager.cc:1367] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.009s	user 0.002s	sys 0.005s
W20240502 08:18:15.780687 14400 leader_election.cc:341] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Illegal state: must be running to vote when last-logged opid is not known
I20240502 08:18:15.780838 14400 leader_election.cc:302] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a; no voters: 8ae1cf63e5bd46cf9a4ba89651971dde, b19bedd6f7f94bed9c727f6c7210cc8a
I20240502 08:18:15.780748 15816 raft_consensus.cc:348] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.781114 15573 raft_consensus.cc:2726] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 08:18:15.780967 15816 raft_consensus.cc:374] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.781422 15816 raft_consensus.cc:729] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.781764 15816 consensus_queue.cc:260] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.782047 15815 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:15.783025 15815 ts_tablet_manager.cc:1367] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.014s	user 0.009s	sys 0.000s
I20240502 08:18:15.783349 15814 raft_consensus.cc:348] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.783550 15814 raft_consensus.cc:374] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.783645 15814 raft_consensus.cc:729] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.783876 15814 consensus_queue.cc:260] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.783970 15815 raft_consensus.cc:348] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.784246 15815 raft_consensus.cc:374] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.784369 15814 ts_tablet_manager.cc:1398] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.000s
I20240502 08:18:15.784370 15815 raft_consensus.cc:729] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.784741 15815 consensus_queue.cc:260] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.785492 15815 ts_tablet_manager.cc:1398] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:15.786298 15816 ts_tablet_manager.cc:1398] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.001s
I20240502 08:18:15.786726 15816 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:15.787070 15573 raft_consensus.cc:482] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.787277 15573 raft_consensus.cc:504] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.787998 15573 leader_election.cc:288] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:15.788898 15818 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:15.789127 15818 ts_tablet_manager.cc:1367] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.021s	user 0.003s	sys 0.004s
I20240502 08:18:15.789467 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eaabbe91524d129a170ad3519c265a" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:15.789697 14336 raft_consensus.cc:2454] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:15.790680 14599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eaabbe91524d129a170ad3519c265a" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:15.790943 14599 raft_consensus.cc:2454] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 0.
I20240502 08:18:15.790814 15818 raft_consensus.cc:348] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.791239 15818 raft_consensus.cc:374] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.791383 15818 raft_consensus.cc:729] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.791695 15818 consensus_queue.cc:260] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.791786 14402 leader_election.cc:302] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde; no voters: 
I20240502 08:18:15.792356 15792 raft_consensus.cc:2781] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.792454 15818 ts_tablet_manager.cc:1398] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:18:15.792469 15792 raft_consensus.cc:482] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.792573 15792 raft_consensus.cc:3037] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.795377 15818 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:15.797971 15818 tablet_bootstrap.cc:654] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.799250 15816 tablet_bootstrap.cc:654] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:15.799335 15792 raft_consensus.cc:504] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.801239 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eaabbe91524d129a170ad3519c265a" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:15.801244 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eaabbe91524d129a170ad3519c265a" candidate_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:15.801447 14336 raft_consensus.cc:3037] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.801470 14600 raft_consensus.cc:3037] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.804148 14336 raft_consensus.cc:2454] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:15.804786 14402 leader_election.cc:302] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde; no voters: 
I20240502 08:18:15.805158 15573 raft_consensus.cc:2781] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.807641 14600 raft_consensus.cc:2454] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32e074cd0dfd4311adc4e2a0f5d1ad8a in term 1.
I20240502 08:18:15.800292 15792 leader_election.cc:288] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:15.817523 15573 raft_consensus.cc:686] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Leader. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:15.818106 15573 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:15.822338 15818 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:15.822185 14062 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67). New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:15.822532 15818 ts_tablet_manager.cc:1367] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.027s	user 0.001s	sys 0.010s
I20240502 08:18:15.823339 15816 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:15.823421 15818 raft_consensus.cc:348] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:15.823657 15818 raft_consensus.cc:374] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.823727 15818 raft_consensus.cc:729] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.823588 15816 ts_tablet_manager.cc:1367] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.037s	user 0.007s	sys 0.005s
I20240502 08:18:15.823976 15818 consensus_queue.cc:260] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:15.824544 15818 ts_tablet_manager.cc:1398] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:15.824918 15816 raft_consensus.cc:348] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.825191 15816 raft_consensus.cc:374] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:15.825325 15816 raft_consensus.cc:729] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:15.825663 15816 consensus_queue.cc:260] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.826454 15816 ts_tablet_manager.cc:1398] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:18:15.916182 15802 raft_consensus.cc:482] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:15.916391 15802 raft_consensus.cc:504] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.917045 15802 leader_election.cc:288] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:15.917500 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1fecce883346459e21c29b98d95b7b" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:15.917763 14468 raft_consensus.cc:2454] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:15.918190 14141 leader_election.cc:302] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:15.918148 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1fecce883346459e21c29b98d95b7b" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:15.918397 14336 raft_consensus.cc:2454] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:15.918473 15802 raft_consensus.cc:2781] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:15.918632 15802 raft_consensus.cc:482] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:15.918776 15802 raft_consensus.cc:3037] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.920821 15802 raft_consensus.cc:504] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.921358 15802 leader_election.cc:288] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:15.921938 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1fecce883346459e21c29b98d95b7b" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:15.921967 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1fecce883346459e21c29b98d95b7b" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:15.922183 14336 raft_consensus.cc:3037] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.922263 14468 raft_consensus.cc:3037] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:15.924093 14336 raft_consensus.cc:2454] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:15.924114 14468 raft_consensus.cc:2454] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:15.924501 14141 leader_election.cc:302] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:15.924772 15802 raft_consensus.cc:2781] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:15.924919 15802 raft_consensus.cc:686] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:15.925165 15802 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:15.928244 14061 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:16.035030 15802 raft_consensus.cc:482] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.035244 15802 raft_consensus.cc:504] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:16.035916 15802 leader_election.cc:288] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:16.036445 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017d794c61345959c52f062dd39ef35" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:16.036540 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017d794c61345959c52f062dd39ef35" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:16.036733 14600 raft_consensus.cc:2454] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:16.036773 14468 raft_consensus.cc:2454] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:16.037336 14141 leader_election.cc:302] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:16.037657 15802 raft_consensus.cc:2781] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.037761 15802 raft_consensus.cc:482] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.037883 15802 raft_consensus.cc:3037] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.039875 15802 raft_consensus.cc:504] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:16.040493 15802 leader_election.cc:288] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:16.040910 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017d794c61345959c52f062dd39ef35" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:16.040917 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017d794c61345959c52f062dd39ef35" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:16.041116 14600 raft_consensus.cc:3037] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.041131 14468 raft_consensus.cc:3037] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.043103 14468 raft_consensus.cc:2454] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:16.043255 14600 raft_consensus.cc:2454] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:16.043577 14141 leader_election.cc:302] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:16.043848 15802 raft_consensus.cc:2781] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:16.044032 15802 raft_consensus.cc:686] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:16.044286 15802 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } }
I20240502 08:18:16.047196 14061 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:16.238817 15573 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.250519 15826 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.448859 15802 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.456120 15802 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:16.481791 15831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:16.482245 15831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:16.490797 15831 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 08:18:16.600205 15453 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:16.600689 15453 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 227.174 ms
  time per row: 0.113587 ms
I20240502 08:18:16.887148 14062 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_8"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:16.896458 14184 tablet_service.cc:1467] Processing CreateTablet for tablet 784f72f28102445f8c4c44a3e3da0d2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.896636 14448 tablet_service.cc:1467] Processing CreateTablet for tablet 784f72f28102445f8c4c44a3e3da0d2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.897029 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784f72f28102445f8c4c44a3e3da0d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.897044 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784f72f28102445f8c4c44a3e3da0d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.896950 14315 tablet_service.cc:1467] Processing CreateTablet for tablet 784f72f28102445f8c4c44a3e3da0d2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.897397 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784f72f28102445f8c4c44a3e3da0d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.898934 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.899309 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dcfec9be9f4f19a43d3f4603ae89d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.903439 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.903568 14447 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee2cdcfd53a469c803f846458dd083b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.903865 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dcfec9be9f4f19a43d3f4603ae89d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.903934 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee2cdcfd53a469c803f846458dd083b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.907210 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee2cdcfd53a469c803f846458dd083b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.907589 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee2cdcfd53a469c803f846458dd083b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.911744 14185 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee2cdcfd53a469c803f846458dd083b (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.912189 14185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee2cdcfd53a469c803f846458dd083b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.913940 15892 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:16.914868 14186 tablet_service.cc:1467] Processing CreateTablet for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:16.915298 14186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dcfec9be9f4f19a43d3f4603ae89d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:16.916654 15892 tablet_bootstrap.cc:654] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.925813 15892 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:16.926107 15892 ts_tablet_manager.cc:1367] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.004s
I20240502 08:18:16.927248 15892 raft_consensus.cc:348] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:16.927539 15892 raft_consensus.cc:374] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.927650 15892 raft_consensus.cc:729] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.927982 15892 consensus_queue.cc:260] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:16.928766 15892 ts_tablet_manager.cc:1398] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:18:16.929122 15892 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:16.930820 15892 tablet_bootstrap.cc:654] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.933779 15895 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:16.936827 15895 tablet_bootstrap.cc:654] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.937791 15893 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:16.938566 15891 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:16.939440 15892 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:16.939651 15892 ts_tablet_manager.cc:1367] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20240502 08:18:16.940352 15893 tablet_bootstrap.cc:654] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.940658 15892 raft_consensus.cc:348] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.940907 15892 raft_consensus.cc:374] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.940994 15892 raft_consensus.cc:729] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.941280 15892 consensus_queue.cc:260] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.941430 15891 tablet_bootstrap.cc:654] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.941929 15892 ts_tablet_manager.cc:1398] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:18:16.944226 15893 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:16.944443 15893 ts_tablet_manager.cc:1367] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.007s	user 0.003s	sys 0.002s
I20240502 08:18:16.945353 15893 raft_consensus.cc:348] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.945621 15893 raft_consensus.cc:374] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.945782 15893 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.946098 15893 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.947707 15893 ts_tablet_manager.cc:1398] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.001s
I20240502 08:18:16.948050 15893 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:16.949141 15891 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:16.949321 15891 ts_tablet_manager.cc:1367] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.000s
I20240502 08:18:16.950165 15891 raft_consensus.cc:348] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:16.950390 15891 raft_consensus.cc:374] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.950481 15891 raft_consensus.cc:729] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.950552 15895 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:16.950805 15895 ts_tablet_manager.cc:1367] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.017s	user 0.007s	sys 0.004s
I20240502 08:18:16.950750 15891 consensus_queue.cc:260] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:16.951342 15893 tablet_bootstrap.cc:654] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.951431 15891 ts_tablet_manager.cc:1398] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:16.951699 15891 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:16.953310 15891 tablet_bootstrap.cc:654] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.953814 15893 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:16.953980 15893 ts_tablet_manager.cc:1367] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.002s
I20240502 08:18:16.953418 15895 raft_consensus.cc:348] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:16.954073 15895 raft_consensus.cc:374] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.954172 15895 raft_consensus.cc:729] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.954434 15895 consensus_queue.cc:260] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:16.954767 15893 raft_consensus.cc:348] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.954977 15893 raft_consensus.cc:374] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.955049 15893 raft_consensus.cc:729] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.955315 15893 consensus_queue.cc:260] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.955948 15893 ts_tablet_manager.cc:1398] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:16.955977 15895 ts_tablet_manager.cc:1398] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.001s
I20240502 08:18:16.956281 15895 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:16.957901 15895 tablet_bootstrap.cc:654] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.960201 15895 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:16.960366 15895 ts_tablet_manager.cc:1367] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:18:16.961196 15895 raft_consensus.cc:348] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.961431 15895 raft_consensus.cc:374] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.961520 15895 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.961768 15895 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.962378 15895 ts_tablet_manager.cc:1398] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:16.962715 15895 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:16.964329 15895 tablet_bootstrap.cc:654] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:16.965282 15891 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:16.965721 15891 ts_tablet_manager.cc:1367] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.005s
I20240502 08:18:16.966583 15891 raft_consensus.cc:348] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.966820 15891 raft_consensus.cc:374] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.966917 15891 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.967015 15895 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:16.967157 15895 ts_tablet_manager.cc:1367] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:18:16.967180 15891 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.967757 15891 ts_tablet_manager.cc:1398] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:16.967885 15895 raft_consensus.cc:348] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.968086 15895 raft_consensus.cc:374] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:16.968189 15895 raft_consensus.cc:729] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:16.968413 15895 consensus_queue.cc:260] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.968945 15895 ts_tablet_manager.cc:1398] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:16.991582 15681 raft_consensus.cc:482] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:16.991825 15681 raft_consensus.cc:504] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.992455 15681 leader_election.cc:288] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:16.992902 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2cdcfd53a469c803f846458dd083b" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:16.992990 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2cdcfd53a469c803f846458dd083b" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:16.993114 14468 raft_consensus.cc:2454] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:16.993202 14205 raft_consensus.cc:2454] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:16.993546 14535 leader_election.cc:302] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:16.993830 15681 raft_consensus.cc:2781] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:16.993932 15681 raft_consensus.cc:482] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:16.994027 15681 raft_consensus.cc:3037] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.995910 15681 raft_consensus.cc:504] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:16.996490 15681 leader_election.cc:288] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:16.996896 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2cdcfd53a469c803f846458dd083b" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:16.996955 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee2cdcfd53a469c803f846458dd083b" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:16.997067 14468 raft_consensus.cc:3037] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.997169 14205 raft_consensus.cc:3037] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:16.999148 14468 raft_consensus.cc:2454] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:16.999222 14205 raft_consensus.cc:2454] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:16.999599 14535 leader_election.cc:302] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:16.999892 15681 raft_consensus.cc:2781] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.000299 15681 raft_consensus.cc:686] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:17.000566 15681 consensus_queue.cc:237] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:17.003551 14061 catalog_manager.cc:5484] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 0 to 1, leader changed from <none> to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.043884 15839 raft_consensus.cc:482] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.044080 15839 raft_consensus.cc:504] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:17.044873 15839 leader_election.cc:288] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:17.045486 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784f72f28102445f8c4c44a3e3da0d2d" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:17.045567 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784f72f28102445f8c4c44a3e3da0d2d" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:17.045766 14468 raft_consensus.cc:2454] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:17.045823 14336 raft_consensus.cc:2454] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:17.046255 14141 leader_election.cc:302] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:17.046554 15839 raft_consensus.cc:2781] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.046726 15839 raft_consensus.cc:482] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.046834 15839 raft_consensus.cc:3037] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.048938 15839 raft_consensus.cc:504] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:17.049652 15839 leader_election.cc:288] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:17.049993 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784f72f28102445f8c4c44a3e3da0d2d" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:17.050232 14468 raft_consensus.cc:3037] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.050166 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "784f72f28102445f8c4c44a3e3da0d2d" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:17.050367 14336 raft_consensus.cc:3037] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.052170 14468 raft_consensus.cc:2454] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:17.052438 14336 raft_consensus.cc:2454] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:17.052584 14141 leader_election.cc:302] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:17.052937 15839 raft_consensus.cc:2781] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.053077 15839 raft_consensus.cc:686] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:17.053366 15839 consensus_queue.cc:237] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:17.056411 14061 catalog_manager.cc:5484] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.159583 15681 raft_consensus.cc:482] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:17.159790 15681 raft_consensus.cc:504] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:17.160507 15681 leader_election.cc:288] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:17.161033 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:17.161142 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" is_pre_election: true
I20240502 08:18:17.161283 14336 raft_consensus.cc:2454] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:17.161422 14205 raft_consensus.cc:2454] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 0.
I20240502 08:18:17.161754 14535 leader_election.cc:302] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:17.162068 15681 raft_consensus.cc:2781] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:17.162184 15681 raft_consensus.cc:482] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:17.162305 15681 raft_consensus.cc:3037] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.164299 15681 raft_consensus.cc:504] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:17.164925 15681 leader_election.cc:288] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Requested vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:17.165369 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:17.165505 14205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:17.165611 14336 raft_consensus.cc:3037] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.165738 14205 raft_consensus.cc:3037] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:17.167655 14336 raft_consensus.cc:2454] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:17.168081 14205 raft_consensus.cc:2454] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 1.
I20240502 08:18:17.168074 14535 leader_election.cc:302] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:17.168387 15681 raft_consensus.cc:2781] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:17.168555 15681 raft_consensus.cc:686] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:17.168871 15681 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } }
I20240502 08:18:17.172472 14061 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 0 to 1, leader changed from <none> to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:17.380647 15681 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.396816 15912 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.526804 15839 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.531273 15917 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:17.599416 15908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:17.599818 15908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:17.607493 15908 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 08:18:17.665117 15912 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:17.672724 15681 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 258.173 ms
  time per row: 0.129087 ms
I20240502 08:18:18.034123 14059 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60652:
name: "rebalance_test_table_9"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:18.043787 14315 tablet_service.cc:1467] Processing CreateTablet for tablet 52331b69903943cb9d74f5ef5787dd00 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.043931 14447 tablet_service.cc:1467] Processing CreateTablet for tablet 52331b69903943cb9d74f5ef5787dd00 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.044317 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52331b69903943cb9d74f5ef5787dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.044327 14447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52331b69903943cb9d74f5ef5787dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.044523 14317 tablet_service.cc:1467] Processing CreateTablet for tablet 6e901787c37a40e9a7c6db44091d11ef (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.044947 14317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e901787c37a40e9a7c6db44091d11ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.045490 14448 tablet_service.cc:1467] Processing CreateTablet for tablet 6e901787c37a40e9a7c6db44091d11ef (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.045938 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e901787c37a40e9a7c6db44091d11ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.051138 14580 tablet_service.cc:1467] Processing CreateTablet for tablet 6e901787c37a40e9a7c6db44091d11ef (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.051599 14580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e901787c37a40e9a7c6db44091d11ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.053022 14315 tablet_service.cc:1467] Processing CreateTablet for tablet cd647b6897bc4292a2dc70ef39abab5e (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.053418 14315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd647b6897bc4292a2dc70ef39abab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.055119 14184 tablet_service.cc:1467] Processing CreateTablet for tablet cd647b6897bc4292a2dc70ef39abab5e (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.055577 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd647b6897bc4292a2dc70ef39abab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.059054 14449 tablet_service.cc:1467] Processing CreateTablet for tablet cd647b6897bc4292a2dc70ef39abab5e (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.059480 14449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd647b6897bc4292a2dc70ef39abab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.060644 15974 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:18.062534 15974 tablet_bootstrap.cc:654] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.062862 15975 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:18.051113 14578 tablet_service.cc:1467] Processing CreateTablet for tablet 52331b69903943cb9d74f5ef5787dd00 (DEFAULT_TABLE table=rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:18.063367 14578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52331b69903943cb9d74f5ef5787dd00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:18.064499 15975 tablet_bootstrap.cc:654] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.065169 15974 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:18.065733 15974 ts_tablet_manager.cc:1367] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.013s	user 0.004s	sys 0.000s
I20240502 08:18:18.066612 15974 raft_consensus.cc:348] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.066870 15975 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:18.067039 15975 ts_tablet_manager.cc:1367] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:18:18.067030 15974 raft_consensus.cc:374] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.067162 15974 raft_consensus.cc:729] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.067472 15974 consensus_queue.cc:260] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.067844 15975 raft_consensus.cc:348] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.068336 15975 raft_consensus.cc:374] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.068459 15975 raft_consensus.cc:729] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.068732 15975 consensus_queue.cc:260] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.070128 15975 ts_tablet_manager.cc:1398] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:18.070421 15975 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:18.072047 15975 tablet_bootstrap.cc:654] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.076730 15974 ts_tablet_manager.cc:1398] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:18:18.077117 15974 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:18.080029 15978 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:18.081996 15978 tablet_bootstrap.cc:654] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.083220 15974 tablet_bootstrap.cc:654] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.086553 15982 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:18.089015 15975 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:18.089252 15975 ts_tablet_manager.cc:1367] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.019s	user 0.007s	sys 0.010s
I20240502 08:18:18.089833 15982 tablet_bootstrap.cc:654] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.090143 15975 raft_consensus.cc:348] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.090476 15975 raft_consensus.cc:374] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.090617 15975 raft_consensus.cc:729] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.091006 15975 consensus_queue.cc:260] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.091235 15978 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:18.091440 15978 ts_tablet_manager.cc:1367] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.012s	user 0.001s	sys 0.010s
I20240502 08:18:18.092365 15975 ts_tablet_manager.cc:1398] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:18.092501 15974 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:18.092715 15975 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:18.092701 15974 ts_tablet_manager.cc:1367] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.016s	user 0.005s	sys 0.002s
I20240502 08:18:18.093551 15974 raft_consensus.cc:348] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.093801 15974 raft_consensus.cc:374] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.093899 15974 raft_consensus.cc:729] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.094357 15975 tablet_bootstrap.cc:654] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.094208 15974 consensus_queue.cc:260] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.094950 15974 ts_tablet_manager.cc:1398] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:18:18.095289 15974 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Bootstrap starting.
I20240502 08:18:18.097146 15974 tablet_bootstrap.cc:654] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.098515 15978 raft_consensus.cc:348] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.098876 15978 raft_consensus.cc:374] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.099027 15978 raft_consensus.cc:729] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.099532 15920 raft_consensus.cc:482] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.099704 15920 raft_consensus.cc:504] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.100440 15920 leader_election.cc:288] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:18.101550 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52331b69903943cb9d74f5ef5787dd00" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:18.101780 14468 raft_consensus.cc:2454] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:18.102214 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52331b69903943cb9d74f5ef5787dd00" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:18.103026 14272 leader_election.cc:302] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde; no voters: 
I20240502 08:18:18.103505 15920 raft_consensus.cc:2781] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.103648 15920 raft_consensus.cc:482] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.103753 15920 raft_consensus.cc:3037] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.107947 15920 raft_consensus.cc:504] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.109381 15975 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: No bootstrap required, opened a new log
I20240502 08:18:18.109318 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52331b69903943cb9d74f5ef5787dd00" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:18.109540 14468 raft_consensus.cc:3037] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.109793 15975 ts_tablet_manager.cc:1367] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.005s
I20240502 08:18:18.110852 15975 raft_consensus.cc:348] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.111174 15974 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: No bootstrap required, opened a new log
I20240502 08:18:18.111204 15975 raft_consensus.cc:374] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.111378 15974 ts_tablet_manager.cc:1367] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent bootstrapping tablet: real 0.016s	user 0.002s	sys 0.010s
I20240502 08:18:18.111405 15975 raft_consensus.cc:729] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.111783 15975 consensus_queue.cc:260] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.112249 15974 raft_consensus.cc:348] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.112461 14468 raft_consensus.cc:2454] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:18.114627 14272 leader_election.cc:302] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde; no voters: 
I20240502 08:18:18.114979 15920 leader_election.cc:288] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:18.115314 15920 raft_consensus.cc:2781] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.115469 15920 raft_consensus.cc:686] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:18.115881 15920 consensus_queue.cc:237] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.112535 15974 raft_consensus.cc:374] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.117116 15974 raft_consensus.cc:729] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.117388 15978 consensus_queue.cc:260] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.118202 15978 ts_tablet_manager.cc:1398] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.027s	user 0.002s	sys 0.000s
I20240502 08:18:18.118436 15974 consensus_queue.cc:260] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.119081 14600 raft_consensus.cc:2454] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:18.119580 15978 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:18.121172 15974 ts_tablet_manager.cc:1398] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.000s
I20240502 08:18:18.123895 15975 ts_tablet_manager.cc:1398] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20240502 08:18:18.121522 15978 tablet_bootstrap.cc:654] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:18.123939 14061 catalog_manager.cc:5484] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.127143 15982 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: No bootstrap required, opened a new log
I20240502 08:18:18.127234 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52331b69903943cb9d74f5ef5787dd00" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:18.127355 15982 ts_tablet_manager.cc:1367] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.042s	user 0.011s	sys 0.011s
I20240502 08:18:18.127470 14600 raft_consensus.cc:3037] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.128453 15982 raft_consensus.cc:348] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.128732 15982 raft_consensus.cc:374] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.128841 15982 raft_consensus.cc:729] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.129149 15982 consensus_queue.cc:260] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.129901 15982 ts_tablet_manager.cc:1398] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:18.129941 14600 raft_consensus.cc:2454] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:18.135161 15978 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: No bootstrap required, opened a new log
I20240502 08:18:18.135393 15978 ts_tablet_manager.cc:1367] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.016s	user 0.004s	sys 0.007s
I20240502 08:18:18.136096 15978 raft_consensus.cc:348] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.136348 15978 raft_consensus.cc:374] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:18.136438 15978 raft_consensus.cc:729] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: FOLLOWER
I20240502 08:18:18.136940 15978 consensus_queue.cc:260] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.137477 15978 ts_tablet_manager.cc:1398] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:18:18.142036 15987 raft_consensus.cc:482] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.142258 15987 raft_consensus.cc:504] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.143114 15987 leader_election.cc:288] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:18.143484 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e901787c37a40e9a7c6db44091d11ef" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:18.143527 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e901787c37a40e9a7c6db44091d11ef" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b" is_pre_election: true
I20240502 08:18:18.143710 14468 raft_consensus.cc:2454] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:18.143741 14600 raft_consensus.cc:2454] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 0.
I20240502 08:18:18.144104 14272 leader_election.cc:302] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:18.144409 15987 raft_consensus.cc:2781] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.144536 15987 raft_consensus.cc:482] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.144629 15987 raft_consensus.cc:3037] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.146876 15987 raft_consensus.cc:504] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.147387 15987 leader_election.cc:288] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:18.147869 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e901787c37a40e9a7c6db44091d11ef" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:18.147881 14600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e901787c37a40e9a7c6db44091d11ef" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:18.148062 14600 raft_consensus.cc:3037] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.148057 14468 raft_consensus.cc:3037] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.149864 14600 raft_consensus.cc:2454] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:18.149904 14468 raft_consensus.cc:2454] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 1.
I20240502 08:18:18.150236 14272 leader_election.cc:302] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ae1cf63e5bd46cf9a4ba89651971dde, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:18.150509 15987 raft_consensus.cc:2781] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.150696 15987 raft_consensus.cc:686] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:18.150976 15987 consensus_queue.cc:237] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }
I20240502 08:18:18.154027 14056 catalog_manager.cc:5484] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.328012 15917 raft_consensus.cc:482] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:18.328229 15917 raft_consensus.cc:504] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.328940 15917 leader_election.cc:288] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:18.329538 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd647b6897bc4292a2dc70ef39abab5e" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:18.329874 14468 raft_consensus.cc:2454] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:18.329914 14337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd647b6897bc4292a2dc70ef39abab5e" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
I20240502 08:18:18.330283 14337 raft_consensus.cc:2454] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 0.
I20240502 08:18:18.330389 14141 leader_election.cc:302] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:18.330727 15917 raft_consensus.cc:2781] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:18:18.330854 15917 raft_consensus.cc:482] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:18:18.330976 15917 raft_consensus.cc:3037] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.333784 15917 raft_consensus.cc:504] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.334493 15917 leader_election.cc:288] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:18.334892 14468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd647b6897bc4292a2dc70ef39abab5e" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:18.335153 14468 raft_consensus.cc:3037] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.335345 14336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd647b6897bc4292a2dc70ef39abab5e" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
I20240502 08:18:18.335546 14336 raft_consensus.cc:3037] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:18.337482 14468 raft_consensus.cc:2454] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:18.337635 14336 raft_consensus.cc:2454] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 1.
I20240502 08:18:18.337934 14141 leader_election.cc:302] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:18.338223 15917 raft_consensus.cc:2781] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:18.338377 15917 raft_consensus.cc:686] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Leader. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
I20240502 08:18:18.338655 15917 consensus_queue.cc:237] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } }
I20240502 08:18:18.341470 14056 catalog_manager.cc:5484] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: term changed from 0 to 1, leader changed from <none> to b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65). New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:18.518222 15987 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.538225 15986 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.572584 15986 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.578043 15986 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:18:18.732559 15992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:18.732970 15992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:18.740765 15992 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 08:18:18.825781 15839 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:18:18.846467 15839 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 277.684 ms
  time per row: 0.138842 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:18:20.134680  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:43257
--local_ip_for_outbound_sockets=127.6.151.69
--tserver_master_addrs=127.6.151.126:35417
--webserver_port=38959
--webserver_interface=127.6.151.69
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:18:20.493978 16055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:20.494277 16055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:20.494415 16055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:20.502081 16055 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:20.502264 16055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:20.502447 16055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:18:20.512383 16055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:43257
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=38959
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:20.513898 16055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:20.515841 16055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:20.528522 16061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:20.528571 16060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:20.529268 16063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:20.529521 16055 server_base.cc:1034] running on GCE node
I20240502 08:18:20.530103 16055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:20.531167 16055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:20.532367 16055 hybrid_clock.cc:648] HybridClock initialized: now 1714637900532309 us; error 54 us; skew 500 ppm
I20240502 08:18:20.532696 16055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:20.534981 16055 webserver.cc:460] Webserver started at http://127.6.151.69:38959/ using document root <none> and password file <none>
I20240502 08:18:20.535753 16055 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:20.535897 16055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:20.541780 16055 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.003s
I20240502 08:18:20.544526 16069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:20.545960 16055 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:20.546128 16055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:20.547003 16055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:20.591394 16055 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:20.592206 16055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:20.592471 16055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:20.593503 16055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:20.595185 16055 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:20.595255 16055 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:20.595352 16055 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:20.595396 16055 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:20.633275 16055 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:43257
I20240502 08:18:20.633360 16182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:43257 every 8 connection(s)
I20240502 08:18:20.634769 16055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:18:20.637457  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 16055
I20240502 08:18:20.638149  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.70:45703
--local_ip_for_outbound_sockets=127.6.151.70
--tserver_master_addrs=127.6.151.126:35417
--webserver_port=40841
--webserver_interface=127.6.151.70
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:20.649435 16183 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:20.649752 16183 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:20.650698 16183 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:20.652428 14056 ts_manager.cc:194] Re-registered known tserver with Master: f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257)
I20240502 08:18:20.653384 14056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:57857
W20240502 08:18:21.013749 16187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:21.014079 16187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:21.014225 16187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:21.022322 16187 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:21.022486 16187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:21.022728 16187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.70
I20240502 08:18:21.032589 16187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.70:45703
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.151.70
--webserver_port=40841
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.70
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:21.033890 16187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:21.035635 16187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:21.048030 16192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:21.048101 16195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:21.048462 16193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:21.049513 16187 server_base.cc:1034] running on GCE node
I20240502 08:18:21.050140 16187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:21.051194 16187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:21.052408 16187 hybrid_clock.cc:648] HybridClock initialized: now 1714637901052365 us; error 40 us; skew 500 ppm
I20240502 08:18:21.052785 16187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:21.055447 16187 webserver.cc:460] Webserver started at http://127.6.151.70:40841/ using document root <none> and password file <none>
I20240502 08:18:21.056365 16187 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:21.056505 16187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:21.062449 16187 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20240502 08:18:21.065466 16201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:21.066988 16187 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:18:21.067162 16187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "5b84b46faddb47fcae86cc5f3f478bf1"
format_stamp: "Formatted at 2024-05-02 08:18:01 on dist-test-slave-x3qn"
I20240502 08:18:21.068002 16187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:21.087489 16187 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:21.088366 16187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:21.088610 16187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:21.089522 16187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:21.091190 16187 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:21.091265 16187 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:21.091359 16187 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:21.091403 16187 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:21.130368 16187 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.70:45703
I20240502 08:18:21.130427 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.70:45703 every 8 connection(s)
I20240502 08:18:21.132071 16187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:18:21.135116  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 16187
I20240502 08:18:21.135872  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.71:39897
--local_ip_for_outbound_sockets=127.6.151.71
--tserver_master_addrs=127.6.151.126:35417
--webserver_port=37309
--webserver_interface=127.6.151.71
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:21.145996 16315 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:21.146368 16315 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:21.147426 16315 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:21.149049 14056 ts_manager.cc:194] Re-registered known tserver with Master: 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703)
I20240502 08:18:21.149845 14056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.70:50593
W20240502 08:18:21.497536 16318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:21.497870 16318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:21.498013 16318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:21.505636 16318 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:21.505790 16318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:21.505967 16318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.71
I20240502 08:18:21.516598 16318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.71:39897
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.151.71
--webserver_port=37309
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.71
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:21.518203 16318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:21.520066 16318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:21.532212 16325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:21.532438 16326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:21.532985 16328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:21.533638 16318 server_base.cc:1034] running on GCE node
I20240502 08:18:21.534227 16318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:21.535228 16318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:21.536432 16318 hybrid_clock.cc:648] HybridClock initialized: now 1714637901536366 us; error 53 us; skew 500 ppm
I20240502 08:18:21.536773 16318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:21.544432 16318 webserver.cc:460] Webserver started at http://127.6.151.71:37309/ using document root <none> and password file <none>
I20240502 08:18:21.545296 16318 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:21.545431 16318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:21.550877 16318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:18:21.553902 16334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:21.555346 16318 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:18:21.555516 16318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b3dfd9d6472342689afd86203f7afe5f"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:21.556334 16318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:21.586709 16318 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:21.587646 16318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:21.587934 16318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:21.588909 16318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:21.590477 16318 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:21.590555 16318 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:21.590713 16318 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:21.590770 16318 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:21.625053 16318 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.71:39897
I20240502 08:18:21.625084 16446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.71:39897 every 8 connection(s)
I20240502 08:18:21.626818 16318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 08:18:21.631493  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 16318
I20240502 08:18:21.632244  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.72:44955
--local_ip_for_outbound_sockets=127.6.151.72
--tserver_master_addrs=127.6.151.126:35417
--webserver_port=38051
--webserver_interface=127.6.151.72
--builtin_ntp_servers=127.6.151.84:42879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:21.644731 16447 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:21.645071 16447 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:21.646162 16447 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:21.647797 14056 ts_manager.cc:194] Re-registered known tserver with Master: b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897)
I20240502 08:18:21.648653 14056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.71:43163
I20240502 08:18:21.655947 16183 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
W20240502 08:18:21.994956 16450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:21.995297 16450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:21.995440 16450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:22.003063 16450 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:22.003201 16450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:22.003377 16450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.72
I20240502 08:18:22.013584 16450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:42879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.72:44955
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.151.72
--webserver_port=38051
--tserver_master_addrs=127.6.151.126:35417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.72
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:22.015213 16450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:22.016916 16450 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:22.028760 16456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:22.028923 16458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:22.029117 16455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:22.030246 16450 server_base.cc:1034] running on GCE node
I20240502 08:18:22.031028 16450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:22.032105 16450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:22.033367 16450 hybrid_clock.cc:648] HybridClock initialized: now 1714637902033299 us; error 37 us; skew 500 ppm
I20240502 08:18:22.033746 16450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:22.036123 16450 webserver.cc:460] Webserver started at http://127.6.151.72:38051/ using document root <none> and password file <none>
I20240502 08:18:22.036890 16450 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:22.037025 16450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:22.042778 16450 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.001s
I20240502 08:18:22.045850 16464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:22.047338 16450 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:18:22.047520 16450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "42116b63c31a436ca060c34691780d04"
format_stamp: "Formatted at 2024-05-02 08:18:02 on dist-test-slave-x3qn"
I20240502 08:18:22.048561 16450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:22.068956 16450 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:22.069795 16450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:22.070080 16450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:22.071096 16450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:22.072672 16450 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:22.072753 16450 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:22.072896 16450 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:22.072960 16450 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:22.108556 16450 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.72:44955
I20240502 08:18:22.108621 16577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.72:44955 every 8 connection(s)
I20240502 08:18:22.110299 16450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 08:18:22.110960  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 16450
I20240502 08:18:22.125378 16578 heartbeater.cc:347] Connected to a master server at 127.6.151.126:35417
I20240502 08:18:22.125682 16578 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:22.126690 16578 heartbeater.cc:511] Master 127.6.151.126:35417 requested a full tablet report, sending...
I20240502 08:18:22.128793 14056 ts_manager.cc:194] Re-registered known tserver with Master: 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955)
I20240502 08:18:22.130116 14056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.72:46027
I20240502 08:18:22.152942 16315 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
I20240502 08:18:22.651789 16447 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
I20240502 08:18:23.135494 16578 heartbeater.cc:503] Master 127.6.151.126:35417 was elected leader, sending a full tablet report...
I20240502 08:18:23.411931 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.450131 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.520504 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.541913 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.544000 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.592424 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.642302 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.644809 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.650082 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:23.673609 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.682308 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.702288 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.759104 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.781144 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.830037 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:23.857616 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.869164 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.906435 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.921921 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:23.928484 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:23.934896 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.946720 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:23.987326 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:24.044767 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:24.263018 14272 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 62341.4us
I20240502 08:18:24.499214 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:24.586467 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:24.615984 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:24.654763 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:24.656576 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:24.699244 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:24.709151 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:24.732121 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:24.753248 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:24.754827 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:24.792297 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:24.847083 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:24.867491 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:24.899098 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:24.917382 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:24.934849 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.273874 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.276227 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:25.280272 14141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 70100.5us
I20240502 08:18:25.310596 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:25.334054 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.350157 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:25.359514 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:25.360014 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.410142 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:25.730731 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.742856 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:25.764003 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.873554 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:25.926215 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.944535 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:25.959179 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.971971 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.984167 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:25.989244 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:26.059952 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.067997 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.089807 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.090616 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.141469 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.182451 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.521590 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.623384 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.666064 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.684690 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.744108 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.746021 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.775631 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.787326 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.807572 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:26.844223 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.848835 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.861905 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.872186 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.880255 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.896843 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:26.911213 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.956158 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:26.983506 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:26.995577 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:27.003340 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:27.009075 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:27.012369 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:27.062927 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:27.087250 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:27.371697 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:27.375532 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:27.442590 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:27.455338 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:27.540349 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:27.546731 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:27.586653 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:27.624044 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240502 08:18:27.678807 14533 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 103153us
I20240502 08:18:27.773222 14337 consensus_queue.cc:237] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:27.787216 14205 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.787961 16809 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.837951 16809 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:27.845074 14205 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:27.852708 14600 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:27.853652 14063 catalog_manager.cc:5484] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:27.855057 17123 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:27.868499 14600 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
W20240502 08:18:27.877211 14272 consensus_peers.cc:485] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96365c4348464db59463182aea8c1c23. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:27.987036 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.008932 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:28.011451 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:28.060606 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.094080 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.098434 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:28.149709 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:28.150965 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.182576 14600 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:28.220712 17302 ts_tablet_manager.cc:902] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:28.257206 17302 tablet_copy_client.cc:308] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:28.275884 14336 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.276571 16638 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.280321 14469 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.281119 16638 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.299508 17239 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:28.301586 14336 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:28.308706 14469 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:28.309980 14062 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:28.322113 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 96365c4348464db59463182aea8c1c23 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:34397)
I20240502 08:18:28.322404 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 96365c4348464db59463182aea8c1c23 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:34397: session id = b3dfd9d6472342689afd86203f7afe5f-96365c4348464db59463182aea8c1c23
I20240502 08:18:28.326500 14357 tablet_copy_source_session.cc:205] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:28.333015 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96365c4348464db59463182aea8c1c23. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:28.345471 14534 consensus_peers.cc:485] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d33a9164ef974a8caae83f3c22b87a51. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:28.426789 17302 tablet_copy_client.cc:783] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:28.427381 17302 tablet_copy_client.cc:647] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:28.431949 17302 tablet_copy_client.cc:523] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:28.436640 17302 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:28.503929 17302 log.cc:824] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:28.609474 17302 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:28.608845 14205 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:28.610456 17302 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:28.611104 17302 ts_tablet_manager.cc:1367] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.175s	user 0.081s	sys 0.005s
I20240502 08:18:28.620157 17302 raft_consensus.cc:348] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:28.621029 17302 raft_consensus.cc:729] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:28.621865 17302 consensus_queue.cc:260] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:28.631971 14336 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.633039 14469 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.633332 17346 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.636307 17346 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.642627 17346 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:28.651021 14336 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:28.662961 14062 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:28.733294 17385 ts_tablet_manager.cc:902] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
W20240502 08:18:28.756721 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c1fecce883346459e21c29b98d95b7b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:28.780259 17302 ts_tablet_manager.cc:1398] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.169s	user 0.055s	sys 0.086s
I20240502 08:18:28.781360 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-96365c4348464db59463182aea8c1c23 received from {username='slave'} at 127.6.151.71:34397
I20240502 08:18:28.781548 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-96365c4348464db59463182aea8c1c23 on tablet 96365c4348464db59463182aea8c1c23 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:28.783445 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:28.792523 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:28.803337 14467 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:28.810494 14469 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:28.822163 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:28.822943 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.841361 14205 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.842159 16902 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.858873 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.870999 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.875803 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:28.882227 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:28.923601 17385 tablet_copy_client.cc:308] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:28.929157 14336 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:28.940382 16902 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:28.959693 14600 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.960528 17123 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.975261 14598 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.978623 14205 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:28.980548 17123 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.987713 17419 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:28.989527 16809 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:28.996150 14598 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:29.007716 14058 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:29.030076 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 53815.3us
I20240502 08:18:29.042348 14204 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:29.045363 14058 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:29.054214 14204 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:29.066900 14598 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:29.068943 14204 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:29.073099 14336 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:29.074609 17382 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:29.074939 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c1fecce883346459e21c29b98d95b7b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.081421 17346 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:29.082672 14469 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:29.082212 14336 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
W20240502 08:18:29.083243 14402 consensus_peers.cc:485] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6272b2f3a9504e01832cb31f5b322980. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.085538 17066 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:29.088057 14270 consensus_peers.cc:485] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf7124419bae40dfaaabdd2e673a3eb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.089349 14056 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:29.091756 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c1fecce883346459e21c29b98d95b7b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.097738 14469 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:29.113304 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:29.135640 16402 raft_consensus.cc:1203] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:29.141726 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:29.178277 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.184852 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.204210 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.206840 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet d33a9164ef974a8caae83f3c22b87a51 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:49697)
I20240502 08:18:29.207204 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.223096 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:29.224997 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet d33a9164ef974a8caae83f3c22b87a51 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:49697: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-d33a9164ef974a8caae83f3c22b87a51
I20240502 08:18:29.228855 14620 tablet_copy_source_session.cc:205] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:29.238888 17385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a9164ef974a8caae83f3c22b87a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.242153 17302 ts_tablet_manager.cc:902] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:29.262287 14598 consensus_queue.cc:237] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:29.267601 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:29.271574 17385 tablet_copy_client.cc:783] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:29.271934 17385 tablet_copy_client.cc:647] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
W20240502 08:18:29.279753 14534 consensus_peers.cc:485] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ee2cdcfd53a469c803f846458dd083b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.281803 14469 raft_consensus.cc:1261] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:29.282764 17239 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.288852 17385 tablet_copy_client.cc:523] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:29.293668 17385 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:29.300522 14204 raft_consensus.cc:1261] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:29.320755 17470 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.321636 17385 log.cc:824] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:29.347748 14467 consensus_queue.cc:237] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:29.379266 17470 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:29.381705 14204 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:29.412540 14336 raft_consensus.cc:1261] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:29.413270 17420 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.414921 14056 catalog_manager.cc:5484] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:29.417402 14401 consensus_peers.cc:485] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44f747cd88e04d23951f351b1578bbb8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.427052 14469 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:29.432034 17479 ts_tablet_manager.cc:902] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:29.437440 14205 raft_consensus.cc:1261] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:29.438992 17420 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.455449 17302 tablet_copy_client.cc:308] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:29.488631 17420 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:29.496551 17493 ts_tablet_manager.cc:902] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:29.501780 14056 catalog_manager.cc:5484] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:29.509778 14337 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:29.530150 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 6272b2f3a9504e01832cb31f5b322980 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:59591)
I20240502 08:18:29.530431 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 6272b2f3a9504e01832cb31f5b322980 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:59591: session id = b3dfd9d6472342689afd86203f7afe5f-6272b2f3a9504e01832cb31f5b322980
I20240502 08:18:29.534893 14489 tablet_copy_source_session.cc:205] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:29.536911 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6272b2f3a9504e01832cb31f5b322980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.541363 17493 tablet_copy_client.cc:308] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:29.549496 17302 tablet_copy_client.cc:783] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:29.549860 17302 tablet_copy_client.cc:647] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:29.552889 17302 tablet_copy_client.cc:523] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:29.560415 17302 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:29.597461 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:29.613282 14204 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:29.616935 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.647976 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:29.658450 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.683516 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 2c1fecce883346459e21c29b98d95b7b from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:50837)
I20240502 08:18:29.683794 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 2c1fecce883346459e21c29b98d95b7b from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:50837: session id = 5b84b46faddb47fcae86cc5f3f478bf1-2c1fecce883346459e21c29b98d95b7b
I20240502 08:18:29.689196 14226 tablet_copy_source_session.cc:205] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:29.695820 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1fecce883346459e21c29b98d95b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.698238 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:29.703866 17479 tablet_copy_client.cc:308] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:29.704488 17515 ts_tablet_manager.cc:902] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:29.705956 17517 ts_tablet_manager.cc:902] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:29.725961 14336 consensus_queue.cc:237] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:29.746553 17517 tablet_copy_client.cc:308] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:29.747560 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 9ee2cdcfd53a469c803f846458dd083b from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:49697)
I20240502 08:18:29.747648 17515 tablet_copy_client.cc:308] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:29.747766 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 9ee2cdcfd53a469c803f846458dd083b from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:49697: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-9ee2cdcfd53a469c803f846458dd083b
I20240502 08:18:29.754043 16809 raft_consensus.cc:1050] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:29.755254 16809 consensus_queue.cc:237] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:29.758064 14620 tablet_copy_source_session.cc:205] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:29.760563 17517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee2cdcfd53a469c803f846458dd083b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.762750 14469 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:29.768002 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet cf7124419bae40dfaaabdd2e673a3eb0 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:57815)
I20240502 08:18:29.768205 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet cf7124419bae40dfaaabdd2e673a3eb0 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:57815: session id = 42116b63c31a436ca060c34691780d04-cf7124419bae40dfaaabdd2e673a3eb0
I20240502 08:18:29.771301 14357 tablet_copy_source_session.cc:205] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:29.776721 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7124419bae40dfaaabdd2e673a3eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.786751 17493 tablet_copy_client.cc:783] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:29.787199 17493 tablet_copy_client.cc:647] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:29.795023 17493 tablet_copy_client.cc:523] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:29.800886 17539 ts_tablet_manager.cc:902] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
W20240502 08:18:29.808600 14402 consensus_peers.cc:485] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a635dc6d64724295be4b4a7f68d0fad6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.808787 14600 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:29.808964 14270 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5341f4bf3c9643859859ed0d9887cfca. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:29.809227 14598 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:29.809429 17123 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.809957 17419 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.811233 14337 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:29.811935 14204 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:29.812896 17123 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.814254 17419 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:29.815109 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 2c1fecce883346459e21c29b98d95b7b from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:40247)
I20240502 08:18:29.815332 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 2c1fecce883346459e21c29b98d95b7b from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:40247: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-2c1fecce883346459e21c29b98d95b7b
I20240502 08:18:29.819778 14226 tablet_copy_source_session.cc:205] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:29.823331 17515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1fecce883346459e21c29b98d95b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:29.827055 16402 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:29.828075 17123 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:29.835475 14599 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:29.836067 16809 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:29.836340 14205 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:29.837669 17123 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:29.839043 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.856379 17493 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:29.860029 17419 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:29.862157 17515 tablet_copy_client.cc:783] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:29.862427 17515 tablet_copy_client.cc:647] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:29.863636 14337 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:29.864473 17515 tablet_copy_client.cc:523] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:29.870750 14065 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:29.885381 17548 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:29.885952 14205 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:29.892408 17515 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:29.893414 14056 catalog_manager.cc:5484] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:29.894224 14599 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:29.897528 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:29.897150 16809 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:29.902570 17517 tablet_copy_client.cc:783] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:29.902966 17517 tablet_copy_client.cc:647] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:29.910178 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:29.916518 17517 tablet_copy_client.cc:523] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:29.929116 17385 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:29.929849 17385 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:29.931440 14204 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:29.936281 17493 log.cc:824] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:29.938475 14056 catalog_manager.cc:5484] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:29.942051 14600 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:29.949047 16402 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:29.956293 17517 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:29.957397 17385 ts_tablet_manager.cc:1367] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.664s	user 0.131s	sys 0.047s
I20240502 08:18:29.963677 14337 consensus_queue.cc:237] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:29.966657 14598 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:29.990631 17385 raft_consensus.cc:348] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:29.991273 17385 raft_consensus.cc:729] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:29.991923 17385 consensus_queue.cc:260] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.008702 14600 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:30.009951 17553 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:30.012080 17385 ts_tablet_manager.cc:1398] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.055s	user 0.010s	sys 0.000s
I20240502 08:18:30.013769 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-d33a9164ef974a8caae83f3c22b87a51 received from {username='slave'} at 127.6.151.69:49697
I20240502 08:18:30.013952 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-d33a9164ef974a8caae83f3c22b87a51 on tablet d33a9164ef974a8caae83f3c22b87a51 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:30.018244 16401 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:30.021368 14204 consensus_queue.cc:237] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:30.025321 17553 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:30.030498 17479 tablet_copy_client.cc:783] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:30.030912 17479 tablet_copy_client.cc:647] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:30.031183 17548 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:30.034344 16402 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:30.036276 14600 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:30.037168 17479 tablet_copy_client.cc:523] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:30.038089 14056 catalog_manager.cc:5484] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:30.039055 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 96365c4348464db59463182aea8c1c23 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:30.043910 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 96365c4348464db59463182aea8c1c23 with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53474
W20240502 08:18:30.049973 14140 consensus_peers.cc:485] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd647b6897bc4292a2dc70ef39abab5e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.051813 14467 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.052539 14337 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.053455 17382 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.054702 17382 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.078609 14336 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:30.079659 17539 tablet_copy_client.cc:308] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:30.083091 17573 tablet_replica.cc:330] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:30.084300 17573 raft_consensus.cc:2229] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:30.086176 17573 raft_consensus.cc:2258] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.088366 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 44f747cd88e04d23951f351b1578bbb8 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:34071)
I20240502 08:18:30.088559 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 44f747cd88e04d23951f351b1578bbb8 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:34071: session id = 5b84b46faddb47fcae86cc5f3f478bf1-44f747cd88e04d23951f351b1578bbb8
I20240502 08:18:30.090138 17573 ts_tablet_manager.cc:1875] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:30.092621 14489 tablet_copy_source_session.cc:205] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:30.092684 14598 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.093295 17123 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:30.094202 14270 consensus_peers.cc:485] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.095372 14272 consensus_peers.cc:485] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf7124419bae40dfaaabdd2e673a3eb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.096320 17539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f747cd88e04d23951f351b1578bbb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.099599 14204 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.100229 17123 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.105883 17449 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.106385 17573 ts_tablet_manager.cc:1888] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:30.106540 17573 log.cc:1196] T 96365c4348464db59463182aea8c1c23 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/96365c4348464db59463182aea8c1c23
I20240502 08:18:30.109202 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet 96365c4348464db59463182aea8c1c23 (table rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]) successfully deleted
I20240502 08:18:30.112840 14056 catalog_manager.cc:5484] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:30.114996 14467 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.115324 14205 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:30.120821 17548 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
W20240502 08:18:30.130007 14140 consensus_peers.cc:485] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1017d794c61345959c52f062dd39ef35. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.139134 17479 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
W20240502 08:18:30.139616 17578 log.cc:924] Time spent T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20240502 08:18:30.140241 14467 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.140292 14600 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.141053 17449 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.141769 17539 tablet_copy_client.cc:783] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:30.141788 17449 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.143496 17539 tablet_copy_client.cc:647] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:30.145901 17539 tablet_copy_client.cc:523] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:30.147699 17302 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:30.148391 17302 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:30.148700 17302 ts_tablet_manager.cc:1367] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.588s	user 0.060s	sys 0.001s
I20240502 08:18:30.148821 17449 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.151719 14337 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.153016 17302 raft_consensus.cc:348] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:30.153851 17302 raft_consensus.cc:729] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:30.154062 17302 consensus_queue.cc:260] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:30.155493 14600 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.164186 17302 ts_tablet_manager.cc:1398] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.015s	user 0.002s	sys 0.001s
I20240502 08:18:30.165997 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-6272b2f3a9504e01832cb31f5b322980 received from {username='slave'} at 127.6.151.71:59591
I20240502 08:18:30.166165 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-6272b2f3a9504e01832cb31f5b322980 on tablet 6272b2f3a9504e01832cb31f5b322980 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:30.168985 17539 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:30.167987 14056 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.175649 16138 raft_consensus.cc:1203] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:30.181562 14205 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:30.184446 14059 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:30.211350 16402 raft_consensus.cc:1203] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:30.211972 14600 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:30.230883 17598 ts_tablet_manager.cc:902] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:30.234737 14469 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:30.237573 14467 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.264751 16402 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.264822 14598 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.265408 17420 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.266330 16902 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.289635 14205 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.290820 16902 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.297729 17493 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:30.298507 17493 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:30.305635 17493 ts_tablet_manager.cc:1367] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.451s	user 0.058s	sys 0.016s
I20240502 08:18:30.309819 17479 log.cc:824] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:30.311211 17420 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.318445 17493 raft_consensus.cc:348] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.319208 17493 raft_consensus.cc:729] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:30.319128 14205 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.319890 17493 consensus_queue.cc:260] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.325923 14058 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
W20240502 08:18:30.327854 17600 log.cc:924] Time spent T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20240502 08:18:30.344413 16402 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.347693 14469 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.354840 17493 ts_tablet_manager.cc:1398] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.049s	user 0.023s	sys 0.010s
I20240502 08:18:30.356572 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-2c1fecce883346459e21c29b98d95b7b received from {username='slave'} at 127.6.151.70:50837
I20240502 08:18:30.356777 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-2c1fecce883346459e21c29b98d95b7b on tablet 2c1fecce883346459e21c29b98d95b7b with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:30.362617 16902 raft_consensus.cc:1050] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:30.369110 16902 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:30.372437 17420 raft_consensus.cc:1013] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: attempt to promote peer b3dfd9d6472342689afd86203f7afe5f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 08:18:30.372565 16402 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:30.372326 14598 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.374485 17627 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:30.376734 14205 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:30.370358 17598 tablet_copy_client.cc:308] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:30.380811 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 5341f4bf3c9643859859ed0d9887cfca from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:57815)
I20240502 08:18:30.380996 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 5341f4bf3c9643859859ed0d9887cfca from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:57815: session id = 42116b63c31a436ca060c34691780d04-5341f4bf3c9643859859ed0d9887cfca
I20240502 08:18:30.384125 14357 tablet_copy_source_session.cc:205] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:30.396441 17302 ts_tablet_manager.cc:902] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:30.397527 17598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5341f4bf3c9643859859ed0d9887cfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.404418 14336 consensus_queue.cc:237] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.406994 14205 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.407006 14599 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.407809 17553 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.408078 17123 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:30.410049 14270 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5341f4bf3c9643859859ed0d9887cfca. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.412657 17553 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:30.415297 14599 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:30.415553 14205 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:30.427091 14056 catalog_manager.cc:5484] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
W20240502 08:18:30.440574 14400 consensus_peers.cc:485] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6272b2f3a9504e01832cb31f5b322980. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.443145 17627 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:30.445788 14597 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.447731 17493 ts_tablet_manager.cc:902] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:30.454886 17420 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.456254 16902 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.463665 16402 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.464087 14205 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.471087 14058 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 5 to 6, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.478204 17419 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:30.485455 14597 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:30.494697 14065 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:30.496665 14205 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.497304 14600 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:30.498093 17420 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.523942 17420 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:30.524118 17658 ts_tablet_manager.cc:902] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:30.526026 17493 tablet_copy_client.cc:308] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:30.527638 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet cd647b6897bc4292a2dc70ef39abab5e from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:50837)
I20240502 08:18:30.527812 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet cd647b6897bc4292a2dc70ef39abab5e from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:50837: session id = 5b84b46faddb47fcae86cc5f3f478bf1-cd647b6897bc4292a2dc70ef39abab5e
I20240502 08:18:30.532353 14226 tablet_copy_source_session.cc:205] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:30.533808 17493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd647b6897bc4292a2dc70ef39abab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.533743 14597 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
W20240502 08:18:30.544903 14271 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5341f4bf3c9643859859ed0d9887cfca. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.548534 14401 consensus_peers.cc:485] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f109afd932d4c77a6b7cd63c417bc70. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.550567 16269 raft_consensus.cc:1203] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 08:18:30.552785 14205 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:30.563573 14597 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:30.567749 17658 tablet_copy_client.cc:308] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:30.568522 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet cf7124419bae40dfaaabdd2e673a3eb0 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:34397)
I20240502 08:18:30.568732 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet cf7124419bae40dfaaabdd2e673a3eb0 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:34397: session id = b3dfd9d6472342689afd86203f7afe5f-cf7124419bae40dfaaabdd2e673a3eb0
I20240502 08:18:30.571851 14357 tablet_copy_source_session.cc:205] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:30.573738 17658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7124419bae40dfaaabdd2e673a3eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.580855 17598 tablet_copy_client.cc:783] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:30.581193 17598 tablet_copy_client.cc:647] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:30.585285 17598 tablet_copy_client.cc:523] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:30.591197 17493 tablet_copy_client.cc:783] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:30.591516 17493 tablet_copy_client.cc:647] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:30.591955 14337 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.592922 14469 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.593411 17493 tablet_copy_client.cc:523] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:30.593925 17470 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.594552 17483 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.594710 17658 tablet_copy_client.cc:783] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:30.594949 17658 tablet_copy_client.cc:647] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:30.597920 17493 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:30.611227 17658 tablet_copy_client.cc:523] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:30.617249 17239 raft_consensus.cc:1050] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:30.618369 17239 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:30.623965 17483 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.627736 14336 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.633188 14056 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:30.645399 17683 ts_tablet_manager.cc:902] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:30.647576 16138 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.649426 14469 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.649808 17483 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.650094 14467 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.661532 17470 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:30.664631 14335 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:30.665196 17685 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:30.672827 14534 consensus_peers.cc:485] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930e3af93722422fb9237fd9b623ceac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.680816 17483 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:30.686864 16138 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:30.686964 14467 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:30.691547 17302 tablet_copy_client.cc:308] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:30.692400 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet a635dc6d64724295be4b4a7f68d0fad6 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:59591)
I20240502 08:18:30.692569 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet a635dc6d64724295be4b4a7f68d0fad6 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:59591: session id = b3dfd9d6472342689afd86203f7afe5f-a635dc6d64724295be4b4a7f68d0fad6
I20240502 08:18:30.695968 14489 tablet_copy_source_session.cc:205] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:30.705034 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a635dc6d64724295be4b4a7f68d0fad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.704941 14205 consensus_queue.cc:237] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:30.706833 17683 tablet_copy_client.cc:308] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:30.707581 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 1017d794c61345959c52f062dd39ef35 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:50837)
I20240502 08:18:30.707773 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 1017d794c61345959c52f062dd39ef35 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:50837: session id = 5b84b46faddb47fcae86cc5f3f478bf1-1017d794c61345959c52f062dd39ef35
I20240502 08:18:30.708817 17598 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:30.708892 14469 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.709658 17449 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.710633 14226 tablet_copy_source_session.cc:205] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 08:18:30.712401 14140 consensus_peers.cc:485] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 784f72f28102445f8c4c44a3e3da0d2d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.712567 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017d794c61345959c52f062dd39ef35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.712359 14337 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:30.713357 17066 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.713989 14469 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.715517 14336 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.716158 17449 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.721769 14065 catalog_manager.cc:5484] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.729750 17515 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:30.730397 17515 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:30.731335 17515 ts_tablet_manager.cc:1367] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.839s	user 0.053s	sys 0.009s
I20240502 08:18:30.732643 17515 raft_consensus.cc:348] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.733017 17515 raft_consensus.cc:729] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:30.733285 17515 consensus_queue.cc:260] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.733564 14056 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:30.734019 14335 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:30.744297 14598 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:30.746732 16138 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:30.748826 17685 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:30.752698 17239 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:30.756713 17683 tablet_copy_client.cc:783] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:30.757061 17515 ts_tablet_manager.cc:1398] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.026s	user 0.002s	sys 0.000s
I20240502 08:18:30.757948 17683 tablet_copy_client.cc:647] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:30.758255 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d33a9164ef974a8caae83f3c22b87a51 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:30.760242 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-2c1fecce883346459e21c29b98d95b7b received from {username='slave'} at 127.6.151.69:40247
I20240502 08:18:30.760380 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-2c1fecce883346459e21c29b98d95b7b on tablet 2c1fecce883346459e21c29b98d95b7b with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:30.761075 14056 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:30.764083 17683 tablet_copy_client.cc:523] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:30.768362 14469 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:30.769536 17239 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:30.771914 14469 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:30.777729 16138 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:30.778710 17658 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:30.782745 17302 tablet_copy_client.cc:783] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:30.790892 17302 tablet_copy_client.cc:647] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:30.796958 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet d33a9164ef974a8caae83f3c22b87a51 with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:30.805146 17683 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:30.810981 14598 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:30.820437 17302 tablet_copy_client.cc:523] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:30.824579 17302 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:30.829228 14205 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:30.834439 14337 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.836318 17449 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.839864 17515 ts_tablet_manager.cc:902] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:30.843751 14469 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.844954 17517 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:30.845430 17517 tablet_bootstrap.cc:492] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:30.845733 17517 ts_tablet_manager.cc:1367] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.915s	user 0.059s	sys 0.004s
I20240502 08:18:30.846495 17517 raft_consensus.cc:348] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.846786 17517 raft_consensus.cc:729] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:30.847018 17517 consensus_queue.cc:260] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:30.848767 17719 ts_tablet_manager.cc:902] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:30.849743 17066 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.849082 14335 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.850993 14205 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:30.846272 17720 tablet_replica.cc:330] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:30.851488 17239 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.852048 17239 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:30.860313 17517 ts_tablet_manager.cc:1398] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I20240502 08:18:30.862069 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-9ee2cdcfd53a469c803f846458dd083b received from {username='slave'} at 127.6.151.69:49697
I20240502 08:18:30.862210 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-9ee2cdcfd53a469c803f846458dd083b on tablet 9ee2cdcfd53a469c803f846458dd083b with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:30.875339 17719 tablet_copy_client.cc:308] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:30.884546 17720 raft_consensus.cc:2229] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:30.888885 17720 raft_consensus.cc:2258] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:30.890275 17685 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.890954 14335 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.897099 17066 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.897828 14059 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.906757 14335 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.914232 14205 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.911094 14059 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:30.923928 17720 ts_tablet_manager.cc:1875] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:30.930855 14139 consensus_peers.cc:485] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0bcdf713f124d46a22ddfc2873cd742. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.936182 14469 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.944921 14467 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
W20240502 08:18:30.948326 14400 consensus_peers.cc:485] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6272b2f3a9504e01832cb31f5b322980. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:30.954313 14533 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76dcfec9be9f4f19a43d3f4603ae89d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:30.955824 14598 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:30.955847 16402 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:30.956954 17420 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:30.957760 17627 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:30.962235 17419 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.966953 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6272b2f3a9504e01832cb31f5b322980 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:30.969185 16402 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.969211 14598 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.972460 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 6272b2f3a9504e01832cb31f5b322980 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:43863)
I20240502 08:18:30.972647 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 6272b2f3a9504e01832cb31f5b322980 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:43863: session id = 42116b63c31a436ca060c34691780d04-6272b2f3a9504e01832cb31f5b322980
I20240502 08:18:30.976388 14489 tablet_copy_source_session.cc:205] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:30.977806 14058 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 6 to 7, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:30.986147 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 6272b2f3a9504e01832cb31f5b322980 with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:30.987967 17517 ts_tablet_manager.cc:902] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:30.991030 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6272b2f3a9504e01832cb31f5b322980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:30.991834 17515 tablet_copy_client.cc:308] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:30.998224 17719 tablet_copy_client.cc:783] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.000299 17719 tablet_copy_client.cc:647] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.003824 17719 tablet_copy_client.cc:523] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.010735 17719 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:31.016160 17720 ts_tablet_manager.cc:1888] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:31.022969 17720 log.cc:1196] T d33a9164ef974a8caae83f3c22b87a51 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/d33a9164ef974a8caae83f3c22b87a51
I20240502 08:18:31.024099 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet d33a9164ef974a8caae83f3c22b87a51 (table rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]) successfully deleted
I20240502 08:18:31.025358 17382 raft_consensus.cc:1050] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:31.026782 17382 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:31.030297 17539 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.033475 17755 tablet_replica.cc:330] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:31.033712 17755 raft_consensus.cc:2229] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:31.034072 17755 raft_consensus.cc:2258] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:31.036916 17755 ts_tablet_manager.cc:1875] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:31.038187 14467 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:31.040972 17539 tablet_bootstrap.cc:492] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:31.041716 17539 ts_tablet_manager.cc:1367] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.873s	user 0.051s	sys 0.009s
I20240502 08:18:31.044991 14335 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:31.045111 14402 consensus_peers.cc:485] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b44cc3d7fc9944c7abf33b4da92b2f6f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.045652 17419 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.048272 16138 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:31.048942 17066 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:31.050043 17539 raft_consensus.cc:348] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:31.050350 17539 raft_consensus.cc:729] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:31.052153 14205 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.050546 17539 consensus_queue.cc:260] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:31.055038 16269 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:31.055756 17066 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:31.055925 17419 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.066200 14337 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:31.067027 17382 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:31.067423 14467 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:31.067924 17539 ts_tablet_manager.cc:1398] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.000s
I20240502 08:18:31.071578 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-44f747cd88e04d23951f351b1578bbb8 received from {username='slave'} at 127.6.151.70:34071
I20240502 08:18:31.071717 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-44f747cd88e04d23951f351b1578bbb8 on tablet 44f747cd88e04d23951f351b1578bbb8 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:31.072432 17382 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:31.080543 14204 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:31.082430 16902 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:31.084707 14598 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:31.086104 17066 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:31.087458 14140 consensus_peers.cc:485] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.088639 17066 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:31.089362 14469 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:31.089068 14598 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:31.089494 14056 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:31.089887 17449 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:31.091296 14469 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:31.096309 14056 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:31.097308 14335 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:31.109871 14205 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:31.116791 17539 ts_tablet_manager.cc:902] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:31.118876 16137 raft_consensus.cc:1203] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:31.121052 17449 raft_consensus.cc:1013] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: attempt to promote peer f28bbf93da0c45eaaae0959e7a2f1e7e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 08:18:31.129956 17066 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.137995 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 5341f4bf3c9643859859ed0d9887cfca from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:39627)
I20240502 08:18:31.138167 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 5341f4bf3c9643859859ed0d9887cfca from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:39627: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-5341f4bf3c9643859859ed0d9887cfca
I20240502 08:18:31.141146 14357 tablet_copy_source_session.cc:205] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.143683 17515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5341f4bf3c9643859859ed0d9887cfca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.150169 14059 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240502 08:18:31.149881 16269 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.154553 17515 tablet_copy_client.cc:783] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.155798 16138 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.154825 17515 tablet_copy_client.cc:647] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.161746 17515 tablet_copy_client.cc:523] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.163492 14467 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.167001 14337 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.169994 17515 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:31.182484 14205 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:31.191627 17539 tablet_copy_client.cc:308] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:31.206842 14204 consensus_queue.cc:237] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:31.217828 17755 ts_tablet_manager.cc:1888] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:31.218004 17755 log.cc:1196] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/6272b2f3a9504e01832cb31f5b322980
I20240502 08:18:31.227252 17598 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.228108 17598 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:31.235574 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet 6272b2f3a9504e01832cb31f5b322980 (table rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]) successfully deleted
I20240502 08:18:31.238111 17598 ts_tablet_manager.cc:1367] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.529s	user 0.059s	sys 0.001s
W20240502 08:18:31.238355 14140 consensus_peers.cc:485] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a92e7e903594d84a59f3ed058a5a3ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.246946 17517 tablet_copy_client.cc:308] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:31.254912 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 4f109afd932d4c77a6b7cd63c417bc70 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:44057)
I20240502 08:18:31.255106 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 4f109afd932d4c77a6b7cd63c417bc70 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:44057: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-4f109afd932d4c77a6b7cd63c417bc70
W20240502 08:18:31.256306 14141 consensus_peers.cc:485] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1017d794c61345959c52f062dd39ef35. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.257651 14335 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:31.258383 14489 tablet_copy_source_session.cc:205] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.258435 17382 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:31.259060 14468 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:31.259511 16268 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:31.260530 14598 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.261135 17382 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:31.261672 17066 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:31.262853 17517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f109afd932d4c77a6b7cd63c417bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.270041 17479 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.270695 17479 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:31.270727 14467 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.271044 17479 ts_tablet_manager.cc:1367] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 1.198s	user 0.068s	sys 0.008s
I20240502 08:18:31.271067 17809 ts_tablet_manager.cc:902] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:31.271462 17066 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.283818 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 930e3af93722422fb9237fd9b623ceac from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:38521)
I20240502 08:18:31.284016 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 930e3af93722422fb9237fd9b623ceac from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:38521: session id = 5b84b46faddb47fcae86cc5f3f478bf1-930e3af93722422fb9237fd9b623ceac
I20240502 08:18:31.284932 17449 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.287393 14620 tablet_copy_source_session.cc:205] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.287070 17382 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.289067 17539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930e3af93722422fb9237fd9b623ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.291945 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c1fecce883346459e21c29b98d95b7b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:31.294374 16269 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.295166 14056 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:31.308144 14467 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.310732 14337 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.311928 14205 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:31.321602 17066 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.323557 17539 tablet_copy_client.cc:783] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.323980 17539 tablet_copy_client.cc:647] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.329602 17539 tablet_copy_client.cc:523] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.336239 17539 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:31.340494 14598 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.286679 17598 raft_consensus.cc:348] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.340840 17479 raft_consensus.cc:348] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.345284 17479 raft_consensus.cc:729] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:31.344610 17598 raft_consensus.cc:729] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:31.346484 17598 consensus_queue.cc:260] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.348968 17517 tablet_copy_client.cc:783] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.349241 17517 tablet_copy_client.cc:647] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.354174 14468 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.359392 14204 consensus_queue.cc:237] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:31.359640 17302 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.347872 17723 ts_tablet_manager.cc:902] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:31.349427 17479 consensus_queue.cc:260] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.369606 17517 tablet_copy_client.cc:523] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 08:18:31.374946 14140 consensus_peers.cc:485] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6379601e2b2d4bdabde2773fcbe06fb4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.376394 14468 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.339087 17809 tablet_copy_client.cc:308] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:31.348171 16268 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:31.379483 17449 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.380174 14598 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.380753 17719 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.352747 14059 catalog_manager.cc:5484] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.383484 14335 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:31.383574 17517 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:31.384291 17719 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:31.384219 17449 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.385452 17719 ts_tablet_manager.cc:1367] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.375s	user 0.058s	sys 0.003s
I20240502 08:18:31.388619 17719 raft_consensus.cc:348] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.392633 17302 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:31.393719 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 784f72f28102445f8c4c44a3e3da0d2d from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:50837)
I20240502 08:18:31.393898 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 784f72f28102445f8c4c44a3e3da0d2d from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:50837: session id = 5b84b46faddb47fcae86cc5f3f478bf1-784f72f28102445f8c4c44a3e3da0d2d
I20240502 08:18:31.395458 17719 raft_consensus.cc:729] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:31.395711 17719 consensus_queue.cc:260] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.396476 17719 ts_tablet_manager.cc:1398] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.000s
I20240502 08:18:31.397152 14226 tablet_copy_source_session.cc:205] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.397511 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-6272b2f3a9504e01832cb31f5b322980 received from {username='slave'} at 127.6.151.72:43863
I20240502 08:18:31.397666 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session 42116b63c31a436ca060c34691780d04-6272b2f3a9504e01832cb31f5b322980 on tablet 6272b2f3a9504e01832cb31f5b322980 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:31.398795 17449 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:31.400789 17449 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:31.401387 17809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784f72f28102445f8c4c44a3e3da0d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.403398 17449 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.406579 16268 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.408975 17809 tablet_copy_client.cc:783] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.409209 17809 tablet_copy_client.cc:647] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.411137 17809 tablet_copy_client.cc:523] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.413069 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c1fecce883346459e21c29b98d95b7b with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:31.416623 17809 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:31.425951 17449 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.428186 14058 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.436811 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1fecce883346459e21c29b98d95b7b with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:31.442349 14335 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.448436 14056 catalog_manager.cc:5484] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:31.449653 14598 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.458922 17844 tablet_replica.cc:330] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:31.459489 17844 raft_consensus.cc:2229] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:31.460072 17844 raft_consensus.cc:2258] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:31.461107 17719 ts_tablet_manager.cc:902] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:31.472761 14205 consensus_queue.cc:237] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:31.491168 17844 ts_tablet_manager.cc:1875] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:31.391809 17598 ts_tablet_manager.cc:1398] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.147s	user 0.005s	sys 0.007s
I20240502 08:18:31.494263 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-5341f4bf3c9643859859ed0d9887cfca received from {username='slave'} at 127.6.151.72:57815
I20240502 08:18:31.392120 17479 ts_tablet_manager.cc:1398] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.121s	user 0.004s	sys 0.001s
I20240502 08:18:31.497403 17302 ts_tablet_manager.cc:1367] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.673s	user 0.060s	sys 0.003s
I20240502 08:18:31.499115 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session 42116b63c31a436ca060c34691780d04-5341f4bf3c9643859859ed0d9887cfca on tablet 5341f4bf3c9643859859ed0d9887cfca with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:31.500097 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-cf7124419bae40dfaaabdd2e673a3eb0 received from {username='slave'} at 127.6.151.72:57815
I20240502 08:18:31.500244 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session 42116b63c31a436ca060c34691780d04-cf7124419bae40dfaaabdd2e673a3eb0 on tablet cf7124419bae40dfaaabdd2e673a3eb0 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:31.504243 17479 tablet_replica.cc:330] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: stopping tablet replica
I20240502 08:18:31.504889 17479 raft_consensus.cc:2229] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:18:31.466269 16268 raft_consensus.cc:1203] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:31.505322 17479 raft_consensus.cc:2258] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:18:31.507113 17479 ts_tablet_manager.cc:1875] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:31.508901 17302 raft_consensus.cc:348] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
W20240502 08:18:31.513298 14140 consensus_peers.cc:485] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b15694b0b284ecebd899522dc69d422. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.511497 14467 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.516577 17302 raft_consensus.cc:729] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:31.517968 16117 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1fecce883346459e21c29b98d95b7b with delete_type TABLET_DATA_TOMBSTONED (TS f28bbf93da0c45eaaae0959e7a2f1e7e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45260
I20240502 08:18:31.520035 14337 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.516930 17302 consensus_queue.cc:260] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:31.527693 17449 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.534399 17719 tablet_copy_client.cc:308] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:31.519918 14468 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.537770 17844 ts_tablet_manager.cc:1888] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:18:31.539048 17844 log.cc:1196] T 2c1fecce883346459e21c29b98d95b7b P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/2c1fecce883346459e21c29b98d95b7b
I20240502 08:18:31.539309 17449 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.541723 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet 2c1fecce883346459e21c29b98d95b7b (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:31.542845 17858 tablet_replica.cc:330] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: stopping tablet replica
I20240502 08:18:31.543601 17858 raft_consensus.cc:2229] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:18:31.549216 17302 ts_tablet_manager.cc:1398] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.041s	user 0.003s	sys 0.000s
I20240502 08:18:31.550060 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-a635dc6d64724295be4b4a7f68d0fad6 received from {username='slave'} at 127.6.151.71:59591
I20240502 08:18:31.550244 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-a635dc6d64724295be4b4a7f68d0fad6 on tablet a635dc6d64724295be4b4a7f68d0fad6 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:31.557793 17302 ts_tablet_manager.cc:902] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:31.558907 16533 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:31.562116 17821 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:31.562863 17479 ts_tablet_manager.cc:1888] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:18:31.564384 14468 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:31.567970 17479 log.cc:1196] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/wals/6272b2f3a9504e01832cb31f5b322980
I20240502 08:18:31.568574 17479 ts_tablet_manager.cc:902] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:31.572190 17858 raft_consensus.cc:2258] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:18:31.572680 14059 catalog_manager.cc:5484] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:31.574262 17858 ts_tablet_manager.cc:1875] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:31.576884 14335 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:31.584719 17723 tablet_copy_client.cc:308] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:31.619496 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet f0bcdf713f124d46a22ddfc2873cd742 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:34765)
I20240502 08:18:31.619731 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet f0bcdf713f124d46a22ddfc2873cd742 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:34765: session id = 42116b63c31a436ca060c34691780d04-f0bcdf713f124d46a22ddfc2873cd742
I20240502 08:18:31.623164 14226 tablet_copy_source_session.cc:205] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.623319 17658 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.624094 17658 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:31.624840 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bcdf713f124d46a22ddfc2873cd742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.631737 17719 tablet_copy_client.cc:783] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.632053 17719 tablet_copy_client.cc:647] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.632063 17479 tablet_copy_client.cc:308] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:31.633229 17658 ts_tablet_manager.cc:1367] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.931s	user 0.058s	sys 0.006s
I20240502 08:18:31.634739 17719 tablet_copy_client.cc:523] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.635597 17658 raft_consensus.cc:348] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:31.638183 17719 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:31.641180 17658 raft_consensus.cc:729] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:31.642374 17658 consensus_queue.cc:260] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:31.644237 17658 ts_tablet_manager.cc:1398] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.009s	user 0.001s	sys 0.001s
I20240502 08:18:31.648592 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-cf7124419bae40dfaaabdd2e673a3eb0 received from {username='slave'} at 127.6.151.71:34397
I20240502 08:18:31.648763 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-cf7124419bae40dfaaabdd2e673a3eb0 on tablet cf7124419bae40dfaaabdd2e673a3eb0 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:31.654817 17658 ts_tablet_manager.cc:902] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:31.656703 17517 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.657219 17517 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:31.657514 17517 ts_tablet_manager.cc:1367] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.274s	user 0.063s	sys 0.026s
I20240502 08:18:31.658450 17517 raft_consensus.cc:348] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:31.660584 17517 raft_consensus.cc:729] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:31.660835 17517 consensus_queue.cc:260] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:31.663709 17517 ts_tablet_manager.cc:1398] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.002s
I20240502 08:18:31.664422 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-4f109afd932d4c77a6b7cd63c417bc70 received from {username='slave'} at 127.6.151.69:44057
I20240502 08:18:31.664599 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-4f109afd932d4c77a6b7cd63c417bc70 on tablet 4f109afd932d4c77a6b7cd63c417bc70 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:31.677932 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 6272b2f3a9504e01832cb31f5b322980 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:43863)
I20240502 08:18:31.678149 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 6272b2f3a9504e01832cb31f5b322980 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:43863: session id = 42116b63c31a436ca060c34691780d04-6272b2f3a9504e01832cb31f5b322980
I20240502 08:18:31.681584 17493 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.682206 17493 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:31.682440 14489 tablet_copy_source_session.cc:205] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.682540 17493 ts_tablet_manager.cc:1367] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 1.085s	user 0.054s	sys 0.008s
I20240502 08:18:31.683506 17493 raft_consensus.cc:348] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:31.685038 17479 ts_tablet_manager.cc:1875] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:31.695775 17479 ts_tablet_manager.cc:1888] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240502 08:18:31.696256 17858 ts_tablet_manager.cc:1888] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:31.696588 17302 tablet_copy_client.cc:308] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:31.697019 17858 log.cc:1196] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/wals/2c1fecce883346459e21c29b98d95b7b
I20240502 08:18:31.697448 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet b44cc3d7fc9944c7abf33b4da92b2f6f from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:59591)
I20240502 08:18:31.697674 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6272b2f3a9504e01832cb31f5b322980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.697716 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet b44cc3d7fc9944c7abf33b4da92b2f6f from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:59591: session id = b3dfd9d6472342689afd86203f7afe5f-b44cc3d7fc9944c7abf33b4da92b2f6f
I20240502 08:18:31.698066 14050 catalog_manager.cc:4830] TS f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): tablet 2c1fecce883346459e21c29b98d95b7b (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:31.698777 17493 raft_consensus.cc:729] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:31.699040 17493 consensus_queue.cc:260] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:31.699709 17493 ts_tablet_manager.cc:1398] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.017s	user 0.001s	sys 0.000s
I20240502 08:18:31.701306 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-cd647b6897bc4292a2dc70ef39abab5e received from {username='slave'} at 127.6.151.70:50837
I20240502 08:18:31.701440 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-cd647b6897bc4292a2dc70ef39abab5e on tablet cd647b6897bc4292a2dc70ef39abab5e with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:31.704548 14489 tablet_copy_source_session.cc:205] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.705015 17479 tablet_copy_client.cc:783] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.705327 17479 tablet_copy_client.cc:647] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.707978 17479 tablet_copy_client.cc:523] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.709203 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44cc3d7fc9944c7abf33b4da92b2f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.711699 17479 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:31.715857 17302 tablet_copy_client.cc:783] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.716091 17302 tablet_copy_client.cc:647] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.725180 17302 tablet_copy_client.cc:523] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.726590 17239 raft_consensus.cc:1050] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:31.727473 17239 consensus_queue.cc:237] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:31.731546 17302 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:31.734026 16533 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:31.740695 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:49117)
I20240502 08:18:31.740883 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:49117: session id = 42116b63c31a436ca060c34691780d04-76dcfec9be9f4f19a43d3f4603ae89d8
I20240502 08:18:31.744117 14620 tablet_copy_source_session.cc:205] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.746742 14598 consensus_queue.cc:237] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.757401 17683 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.770696 17683 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:31.770682 17723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dcfec9be9f4f19a43d3f4603ae89d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.771085 17683 ts_tablet_manager.cc:1367] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.966s	user 0.050s	sys 0.007s
I20240502 08:18:31.771943 17683 raft_consensus.cc:348] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:31.772305 17683 raft_consensus.cc:729] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:31.772538 17683 consensus_queue.cc:260] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:31.773278 17683 ts_tablet_manager.cc:1398] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:18:31.774221 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-1017d794c61345959c52f062dd39ef35 received from {username='slave'} at 127.6.151.70:50837
I20240502 08:18:31.774361 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-1017d794c61345959c52f062dd39ef35 on tablet 1017d794c61345959c52f062dd39ef35 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:31.792438 16402 raft_consensus.cc:1203] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:31.792634 17723 tablet_copy_client.cc:783] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.792945 17723 tablet_copy_client.cc:647] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.804797 14468 raft_consensus.cc:1261] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:31.805675 17897 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:31.806166 14205 raft_consensus.cc:1261] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:31.808454 17897 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:31.810843 14335 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.811697 17517 ts_tablet_manager.cc:902] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:31.811857 14468 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:31.812494 17897 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:31.813129 17897 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:31.820591 14533 consensus_peers.cc:485] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 810c5f8a81ba4e0ab06129c0178f12e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:31.839933 17517 tablet_copy_client.cc:308] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:31.840715 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 6379601e2b2d4bdabde2773fcbe06fb4 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:40247)
I20240502 08:18:31.840893 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 6379601e2b2d4bdabde2773fcbe06fb4 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:40247: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-6379601e2b2d4bdabde2773fcbe06fb4
I20240502 08:18:31.845052 14226 tablet_copy_source_session.cc:205] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.846803 17517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6379601e2b2d4bdabde2773fcbe06fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.867978 16268 raft_consensus.cc:1203] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:31.868748 16402 raft_consensus.cc:1203] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 08:18:31.870102 17517 tablet_copy_client.cc:783] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.870363 17517 tablet_copy_client.cc:647] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.883824 16137 raft_consensus.cc:1261] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:31.884707 17723 tablet_copy_client.cc:523] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.886716 17517 tablet_copy_client.cc:523] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.886961 17897 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:31.889827 17903 log.cc:924] Time spent T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20240502 08:18:31.890269 17517 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:31.890475 17683 ts_tablet_manager.cc:902] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:31.898581 16902 raft_consensus.cc:1050] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:31.899953 16902 consensus_queue.cc:237] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:31.903962 14337 raft_consensus.cc:1261] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:31.905718 17419 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:31.908352 14204 raft_consensus.cc:1261] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:31.909860 17627 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:31.915139 16902 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.919680 14204 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.921638 17479 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:31.928057 14056 catalog_manager.cc:5484] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.933223 17683 tablet_copy_client.cc:308] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:31.933904 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 0b15694b0b284ecebd899522dc69d422 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:50837)
I20240502 08:18:31.934105 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 0b15694b0b284ecebd899522dc69d422 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:50837: session id = 5b84b46faddb47fcae86cc5f3f478bf1-0b15694b0b284ecebd899522dc69d422
I20240502 08:18:31.936053 14337 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:31.937229 14226 tablet_copy_source_session.cc:205] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:31.938499 17683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b15694b0b284ecebd899522dc69d422. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:31.932327 17479 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:31.947077 17479 ts_tablet_manager.cc:1367] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.235s	user 0.061s	sys 0.004s
I20240502 08:18:31.952207 17479 raft_consensus.cc:348] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.952505 17479 raft_consensus.cc:729] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:31.956947 17479 consensus_queue.cc:260] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:31.961696 17897 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:31.966387 17479 ts_tablet_manager.cc:1398] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.019s	user 0.002s	sys 0.000s
I20240502 08:18:31.967198 17897 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:31.973345 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-6272b2f3a9504e01832cb31f5b322980 received from {username='slave'} at 127.6.151.72:43863
I20240502 08:18:31.974236 17683 tablet_copy_client.cc:783] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:31.978349 17548 raft_consensus.cc:1050] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:31.979142 17683 tablet_copy_client.cc:647] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:31.979583 17548 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:31.981256 17683 tablet_copy_client.cc:523] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:31.983325 16402 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:31.984381 14204 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:31.984371 17817 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:31.985113 17817 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:31.986625 16533 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:31.987367 17553 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:31.994721 14059 catalog_manager.cc:5484] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:31.998219 14467 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:31.999024 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session 42116b63c31a436ca060c34691780d04-6272b2f3a9504e01832cb31f5b322980 on tablet 6272b2f3a9504e01832cb31f5b322980 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:32.010591 17553 raft_consensus.cc:1013] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: attempt to promote peer b3dfd9d6472342689afd86203f7afe5f: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 08:18:32.035485 17683 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:32.035678 17817 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.040220 14206 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.043195 16402 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.046165 17809 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:32.046655 17809 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:32.046948 17809 ts_tablet_manager.cc:1367] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.630s	user 0.054s	sys 0.001s
I20240502 08:18:32.047797 17809 raft_consensus.cc:348] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:32.048053 17809 raft_consensus.cc:729] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:32.048578 14598 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.048791 16532 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.049207 17548 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.056385 17539 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:32.058496 17548 raft_consensus.cc:1050] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:32.055274 14059 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.059866 17548 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:32.063138 17539 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:32.063472 17539 ts_tablet_manager.cc:1367] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.727s	user 0.053s	sys 0.008s
I20240502 08:18:32.064107 17809 consensus_queue.cc:260] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:32.064719 17809 ts_tablet_manager.cc:1398] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.018s	user 0.002s	sys 0.000s
I20240502 08:18:32.065397 17539 raft_consensus.cc:348] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:32.065659 17539 raft_consensus.cc:729] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:32.065922 17539 consensus_queue.cc:260] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:32.067276 14205 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.071195 16268 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.076929 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-784f72f28102445f8c4c44a3e3da0d2d received from {username='slave'} at 127.6.151.70:50837
I20240502 08:18:32.077076 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-784f72f28102445f8c4c44a3e3da0d2d on tablet 784f72f28102445f8c4c44a3e3da0d2d with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:32.066438 17539 ts_tablet_manager.cc:1398] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:18:32.082537 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-930e3af93722422fb9237fd9b623ceac received from {username='slave'} at 127.6.151.70:38521
I20240502 08:18:32.082713 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-930e3af93722422fb9237fd9b623ceac on tablet 930e3af93722422fb9237fd9b623ceac with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:32.090152 17920 ts_tablet_manager.cc:902] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:32.092909 16532 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.094067 17553 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:32.094758 16402 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.097478 17553 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:32.100549 14204 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.101308 17817 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:32.104975 14206 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.104930 17553 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.106148 17723 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:32.111337 16532 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.113838 14058 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 6 to 7, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.126056 16402 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.085950 14335 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:32.064920 14468 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:32.135174 14059 catalog_manager.cc:5484] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:32.137486 17517 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:32.137959 17517 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:32.138340 17517 ts_tablet_manager.cc:1367] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.248s	user 0.045s	sys 0.008s
I20240502 08:18:32.147091 16268 raft_consensus.cc:1203] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:32.160986 17517 raft_consensus.cc:348] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:32.161363 17517 raft_consensus.cc:729] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:32.164068 14596 consensus_queue.cc:237] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:32.164477 17517 consensus_queue.cc:260] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
W20240502 08:18:32.166266 14401 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:45703, user_credentials={real_user=slave}} blocked reactor thread for 268246us
W20240502 08:18:32.170076 14533 consensus_peers.cc:485] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6d3c011ad9546e3b69e23cbef587dbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.170626 14337 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:32.171108 14206 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:32.171355 17239 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:32.171978 17239 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:32.173609 16138 raft_consensus.cc:1203] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:32.179852 16268 raft_consensus.cc:1203] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:32.180050 16269 raft_consensus.cc:1261] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.184720 17627 raft_consensus.cc:1050] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:32.185271 14337 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:32.185657 17627 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:32.188162 16402 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:32.189637 16533 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:32.190286 17548 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:32.191103 17548 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:32.193840 14204 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:32.195472 17817 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:32.195974 17548 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.197441 14204 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.199733 16533 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.204552 17517 ts_tablet_manager.cc:1398] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.066s	user 0.008s	sys 0.002s
I20240502 08:18:32.205323 14056 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.205895 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-6379601e2b2d4bdabde2773fcbe06fb4 received from {username='slave'} at 127.6.151.69:40247
I20240502 08:18:32.206013 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-6379601e2b2d4bdabde2773fcbe06fb4 on tablet 6379601e2b2d4bdabde2773fcbe06fb4 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:32.207350 17817 raft_consensus.cc:1050] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:32.208258 17817 consensus_queue.cc:237] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:32.211683 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf7124419bae40dfaaabdd2e673a3eb0 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:32.220376 16402 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.226959 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7124419bae40dfaaabdd2e673a3eb0 with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:32.238312 17719 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:32.239557 14204 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.239811 14596 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 08:18:32.240375 14271 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.240943 17948 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.246670 17949 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.241472 16533 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.250072 17949 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.252440 17719 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:32.257009 17719 ts_tablet_manager.cc:1367] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.619s	user 0.045s	sys 0.010s
I20240502 08:18:32.260170 17719 raft_consensus.cc:348] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:32.266763 17719 raft_consensus.cc:729] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:32.268337 17948 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:32.268575 14203 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:32.267392 17719 consensus_queue.cc:260] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:32.270680 17719 ts_tablet_manager.cc:1398] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.001s
I20240502 08:18:32.273396 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-f0bcdf713f124d46a22ddfc2873cd742 received from {username='slave'} at 127.6.151.72:34765
I20240502 08:18:32.273556 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 42116b63c31a436ca060c34691780d04-f0bcdf713f124d46a22ddfc2873cd742 on tablet f0bcdf713f124d46a22ddfc2873cd742 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:32.278559 16533 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:32.283886 14056 catalog_manager.cc:5484] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:32.291363 14597 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.296489 17658 tablet_copy_client.cc:308] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:32.296578 16135 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.325798 16533 raft_consensus.cc:1203] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:32.327489 14337 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:32.331835 16402 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:32.332621 17948 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:32.333544 16533 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:32.334911 17948 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:32.337682 17948 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.340484 16401 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.342052 16532 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.343389 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf7124419bae40dfaaabdd2e673a3eb0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:32.347757 14062 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.348270 16135 raft_consensus.cc:1203] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:32.354995 17719 ts_tablet_manager.cc:902] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:32.358094 17239 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:32.365293 14337 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:32.374307 14056 catalog_manager.cc:5484] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:32.396672 17627 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
W20240502 08:18:32.406662 17302 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.064s	sys 0.132s
W20240502 08:18:32.406821 17302 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.064s	sys 0.132s
I20240502 08:18:32.416568 16269 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.424891 17382 raft_consensus.cc:1050] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:32.425778 17382 consensus_queue.cc:237] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:32.432451 14206 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:32.433384 16533 raft_consensus.cc:1203] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:32.441865 14599 consensus_queue.cc:237] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:32.447046 14469 raft_consensus.cc:1261] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.447952 17239 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.451411 17821 raft_consensus.cc:1050] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:32.452366 17239 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.452594 17821 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:32.457104 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ee2cdcfd53a469c803f846458dd083b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:32.458580 14469 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.467856 14062 catalog_manager.cc:5484] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:32.470094 16138 raft_consensus.cc:1261] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.472004 17239 consensus_queue.cc:1035] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.475000 16135 raft_consensus.cc:2932] T 9ee2cdcfd53a469c803f846458dd083b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.481976 16269 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.482815 17449 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.484894 14596 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
W20240502 08:18:32.486567 14141 consensus_peers.cc:485] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1017d794c61345959c52f062dd39ef35. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.486853 14336 consensus_queue.cc:237] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:32.496460 14599 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.499531 17821 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.500064 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:18:32.508247 14271 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.509490 16533 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.510347 17817 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:18:32.513250 14400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 186552us
I20240502 08:18:32.513914 17817 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
W20240502 08:18:32.516180 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.71:39897, user_credentials={real_user=slave}} blocked reactor thread for 203226us
I20240502 08:18:32.517248 16401 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.517513 14600 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.519526 14467 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.519555 16533 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:32.520520 16902 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.520781 17449 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.521242 17960 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.524552 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5341f4bf3c9643859859ed0d9887cfca with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:32.525332 14337 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.534740 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7124419bae40dfaaabdd2e673a3eb0 with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:32.534842 14062 catalog_manager.cc:5484] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:32.537910 14203 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.538514 17948 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:32.539119 17449 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.540304 14203 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:32.541332 14596 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.545475 14185 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ee2cdcfd53a469c803f846458dd083b with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:32.551280 17419 raft_consensus.cc:1050] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:32.552326 17419 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:32.558898 14056 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.563799 14598 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.563958 17951 tablet_replica.cc:330] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:32.564482 17951 raft_consensus.cc:2229] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:32.565037 17951 raft_consensus.cc:2258] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:32.570852 17951 ts_tablet_manager.cc:1875] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:32.573895 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 5341f4bf3c9643859859ed0d9887cfca with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:32.580252 17951 ts_tablet_manager.cc:1888] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:32.582676 17719 tablet_copy_client.cc:308] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:32.592175 17515 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:32.584086 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 810c5f8a81ba4e0ab06129c0178f12e7 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:49117)
I20240502 08:18:32.600042 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 810c5f8a81ba4e0ab06129c0178f12e7 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:49117: session id = 42116b63c31a436ca060c34691780d04-810c5f8a81ba4e0ab06129c0178f12e7
I20240502 08:18:32.603457 14620 tablet_copy_source_session.cc:205] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:32.605006 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810c5f8a81ba4e0ab06129c0178f12e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.609699 17989 tablet_replica.cc:330] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:32.610041 17989 raft_consensus.cc:2229] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:32.610800 17989 raft_consensus.cc:2258] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:32.610904 17951 log.cc:1196] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf7124419bae40dfaaabdd2e673a3eb0
I20240502 08:18:32.612651 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet cf7124419bae40dfaaabdd2e673a3eb0 (table rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]) successfully deleted
I20240502 08:18:32.613379 17951 tablet_replica.cc:330] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:32.613704 17951 raft_consensus.cc:2229] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:32.614199 17951 raft_consensus.cc:2258] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:32.615468 17951 ts_tablet_manager.cc:1875] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:32.617389 17960 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.630971 17982 raft_consensus.cc:1050] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
W20240502 08:18:32.640205 14141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 79478.9us
I20240502 08:18:32.643007 14597 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.646494 14203 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:32.647010 17960 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.650813 14206 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.651530 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 1017d794c61345959c52f062dd39ef35 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:52741)
I20240502 08:18:32.654554 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 1017d794c61345959c52f062dd39ef35 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:52741: session id = b3dfd9d6472342689afd86203f7afe5f-1017d794c61345959c52f062dd39ef35
I20240502 08:18:32.655591 14596 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.658478 14226 tablet_copy_source_session.cc:205] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 08:18:32.659376 14141 consensus_peers.cc:485] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1017d794c61345959c52f062dd39ef35. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:32.659719 17991 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:32.660629 17658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1017d794c61345959c52f062dd39ef35. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:32.665055 14140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:45703, user_credentials={real_user=slave}} blocked reactor thread for 84038.1us
I20240502 08:18:32.666285 17982 consensus_queue.cc:237] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:32.668236 17658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.6.151.65:49557 (request call id 1) took 1014 ms. Trace:
I20240502 08:18:32.668844 17961 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.671069 14336 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.671213 17962 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.674670 17993 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.653352 17989 ts_tablet_manager.cc:1875] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:32.678081 14065 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:32.681370 14467 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.681241 14468 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.682711 17515 tablet_bootstrap.cc:492] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:32.682456 14597 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.683056 17515 ts_tablet_manager.cc:1367] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 1.513s	user 0.046s	sys 0.018s
I20240502 08:18:32.620388 17920 tablet_copy_client.cc:308] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:32.638439 16135 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.683904 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 7a92e7e903594d84a59f3ed058a5a3ec from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:40247)
I20240502 08:18:32.684077 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 7a92e7e903594d84a59f3ed058a5a3ec from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:40247: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-7a92e7e903594d84a59f3ed058a5a3ec
I20240502 08:18:32.688807 14226 tablet_copy_source_session.cc:205] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:32.695292 17920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a92e7e903594d84a59f3ed058a5a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:32.702824 14062 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 6 to 7, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.703195 16269 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.703725 14336 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.704483 17985 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.705237 17985 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.706588 17419 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.708799 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 1017d794c61345959c52f062dd39ef35 with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:32.716647 16269 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.719000 16268 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:32.720084 14597 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.720471 14336 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.723259 16268 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.723063 16402 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:32.725651 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1017d794c61345959c52f062dd39ef35 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:32.726723 17990 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.727473 17990 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:32.728255 17990 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.730437 17995 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.675205 17719 tablet_copy_client.cc:783] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:32.730844 17719 tablet_copy_client.cc:647] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:32.737048 17951 ts_tablet_manager.cc:1888] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:32.737255 17951 log.cc:1196] T 5341f4bf3c9643859859ed0d9887cfca P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/5341f4bf3c9643859859ed0d9887cfca
I20240502 08:18:32.738171 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 5341f4bf3c9643859859ed0d9887cfca (table rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]) successfully deleted
I20240502 08:18:32.742103 17515 raft_consensus.cc:348] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:32.742700 17515 raft_consensus.cc:729] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:32.743036 17515 consensus_queue.cc:260] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:32.743878 17515 ts_tablet_manager.cc:1398] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.061s	user 0.000s	sys 0.002s
I20240502 08:18:32.743001 16269 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:32.744611 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-5341f4bf3c9643859859ed0d9887cfca received from {username='slave'} at 127.6.151.69:39627
I20240502 08:18:32.744778 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-5341f4bf3c9643859859ed0d9887cfca on tablet 5341f4bf3c9643859859ed0d9887cfca with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:32.745151 17960 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.749935 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:32.755448 16135 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.756808 17697 raft_consensus.cc:1050] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:32.757644 17697 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:32.770390 14065 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.771534 16902 raft_consensus.cc:1050] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:32.773128 16902 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:32.775718 14203 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.781252 17985 raft_consensus.cc:1050] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:32.782168 17985 consensus_queue.cc:237] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:32.786425 18019 tablet_replica.cc:330] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:32.788700 14468 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.668452 17658 rpcz_store.cc:276] 0502 08:18:31.653906 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 08:18:31.654054 (+   148us) service_pool.cc:225] Handling call
0502 08:18:31.654974 (+   920us) ts_tablet_manager.cc:903] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
0502 08:18:32.660052 (+1005078us) tablet_copy_client.cc:353] Tablet copy session begun
0502 08:18:32.664752 (+  4700us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:32.664881 (+   129us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 08:18:32.667837 (+  2956us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 08:18:32.667957 (+   120us) tablet_replica.cc:205] Creating consensus instance
0502 08:18:32.668077 (+   120us) ts_tablet_manager.cc:958] Replica registered.
0502 08:18:32.668159 (+    82us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":107,"tablet-copy.queue_time_us":495,"thread_start_us":641048,"threads_started":4}
I20240502 08:18:32.789387 17658 tablet_copy_client.cc:783] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:32.789667 17658 tablet_copy_client.cc:647] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:32.791724 18019 raft_consensus.cc:2229] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:32.792100 18019 raft_consensus.cc:2258] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:32.795086 17982 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.799698 17658 tablet_copy_client.cc:523] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:32.806504 16138 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.807201 17982 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.812901 14597 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.813768 17982 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.823626 18019 ts_tablet_manager.cc:1875] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:32.850736 17985 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.855180 14336 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.860301 17719 tablet_copy_client.cc:523] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:32.868376 16268 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.879962 14058 catalog_manager.cc:5484] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:32.881294 17995 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.886083 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:32.887722 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:32.890784 14335 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.899092 16268 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.899950 17239 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.900679 17239 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:32.903723 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.66:44269, user_credentials={real_user=slave}} blocked reactor thread for 108557us
W20240502 08:18:32.903959 14400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 45524.7us
I20240502 08:18:32.905100 14466 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.905560 14597 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:32.905869 16401 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:32.906391 17993 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.906417 14598 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:32.911768 14467 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:32.912977 18019 ts_tablet_manager.cc:1888] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:32.913183 18019 log.cc:1196] T 1017d794c61345959c52f062dd39ef35 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/1017d794c61345959c52f062dd39ef35
I20240502 08:18:32.914027 17911 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.914422 14058 catalog_manager.cc:5484] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:32.914775 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet 1017d794c61345959c52f062dd39ef35 (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:32.928200 14337 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.921620 16269 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.931932 14206 consensus_queue.cc:237] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:32.936902 16267 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.938858 17995 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.942181 17961 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:32.942294 17982 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.947881 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd647b6897bc4292a2dc70ef39abab5e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:32.951318 17961 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:32.962100 14468 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.969413 16268 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:32.974900 16135 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:32.975898 17939 consensus_queue.cc:1037] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:32.977587 14056 catalog_manager.cc:5484] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:32.978358 16135 raft_consensus.cc:858] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
W20240502 08:18:32.978502 16135 raft_consensus.cc:1565] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }: 
  New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:32.980070 17991 raft_consensus.cc:1050] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:32.981132 17991 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:32.988389 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet cd647b6897bc4292a2dc70ef39abab5e with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:32.996160 14467 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.007505 14467 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:33.007838 17993 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.008248 14467 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.008414 17993 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.016850 14598 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.018028 17302 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:33.020622 17302 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:33.020953 17302 ts_tablet_manager.cc:1367] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 1.290s	user 0.117s	sys 0.134s
I20240502 08:18:33.026998 17302 raft_consensus.cc:348] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:33.028183 17302 raft_consensus.cc:729] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:33.028439 17302 consensus_queue.cc:260] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:33.031589 17302 ts_tablet_manager.cc:1398] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.000s
I20240502 08:18:33.033605 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-b44cc3d7fc9944c7abf33b4da92b2f6f received from {username='slave'} at 127.6.151.71:59591
I20240502 08:18:33.033767 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-b44cc3d7fc9944c7abf33b4da92b2f6f on tablet b44cc3d7fc9944c7abf33b4da92b2f6f with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:33.034632 14467 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.038880 17995 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:33.039949 14468 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.041787 17683 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:33.043900 17683 tablet_bootstrap.cc:492] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:33.044204 17683 ts_tablet_manager.cc:1367] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 1.009s	user 0.089s	sys 0.064s
I20240502 08:18:33.045213 17683 raft_consensus.cc:348] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:33.045472 17683 raft_consensus.cc:729] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:33.045725 17683 consensus_queue.cc:260] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:33.046289 17683 ts_tablet_manager.cc:1398] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:18:33.049810 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-0b15694b0b284ecebd899522dc69d422 received from {username='slave'} at 127.6.151.70:50837
I20240502 08:18:33.050007 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-0b15694b0b284ecebd899522dc69d422 on tablet 0b15694b0b284ecebd899522dc69d422 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:33.052964 16135 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:33.062831 14062 catalog_manager.cc:5484] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.071738 17627 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.074208 16138 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.079391 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:33.081341 17989 ts_tablet_manager.cc:1888] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:33.082561 16402 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.088711 18055 tablet_replica.cc:330] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:33.088956 18055 raft_consensus.cc:2229] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.089226 18055 raft_consensus.cc:2258] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.093318 14058 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 7 to 8, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:33.100553 17948 raft_consensus.cc:1050] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:33.101434 17948 consensus_queue.cc:237] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:33.101675 16138 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:33.105619 14467 consensus_queue.cc:237] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:33.106179 16137 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:33.107306 17939 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:33.107223 14466 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:33.113389 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.113999 18055 ts_tablet_manager.cc:1875] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.126241 17989 log.cc:1196] T cf7124419bae40dfaaabdd2e673a3eb0 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf7124419bae40dfaaabdd2e673a3eb0
I20240502 08:18:33.135471 17989 tablet_replica.cc:330] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:33.135721 17989 raft_consensus.cc:2229] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.135985 17989 raft_consensus.cc:2258] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.140718 14596 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
W20240502 08:18:33.142827 14535 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.67:39913, user_credentials={real_user=slave}} blocked reactor thread for 229159us
I20240502 08:18:33.145462 18055 ts_tablet_manager.cc:1888] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:33.145462 17911 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.145987 18055 log.cc:1196] T cd647b6897bc4292a2dc70ef39abab5e P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/cd647b6897bc4292a2dc70ef39abab5e
I20240502 08:18:33.149328 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet cd647b6897bc4292a2dc70ef39abab5e (table rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]) successfully deleted
I20240502 08:18:33.154172 14334 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.158090 14062 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:33.175724 16268 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.178817 14469 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
W20240502 08:18:33.179898 14401 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84eaabbe91524d129a170ad3519c265a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:33.193460 14203 raft_consensus.cc:1261] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.194423 17419 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.139787 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet cf7124419bae40dfaaabdd2e673a3eb0 (table rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]) successfully deleted
I20240502 08:18:33.205794 17658 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
W20240502 08:18:33.206135 14049 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 66332.9us
I20240502 08:18:33.192368 16269 raft_consensus.cc:1261] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.212430 14337 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:33.219946 17920 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.6.151.65:50739 (request call id 10) took 1364 ms. Trace:
I20240502 08:18:33.224558 14337 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:33.225335 14203 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:33.232220 17985 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:33.233942 14596 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:33.239248 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:32.921262 16533 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:33.004037 16532 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:33.217084 16531 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:33.243623 17985 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:32.874905 18024 ts_tablet_manager.cc:902] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:33.259011 17723 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:33.259595 17723 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:33.259954 17723 ts_tablet_manager.cc:1367] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 1.154s	user 0.053s	sys 0.000s
I20240502 08:18:32.866044 17719 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
W20240502 08:18:33.261848 14401 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:45703, user_credentials={real_user=slave}} blocked reactor thread for 50195.2us
I20240502 08:18:33.262917 17961 consensus_queue.cc:1035] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.263845 17961 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:33.267585 16902 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.272827 16269 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.275346 16267 raft_consensus.cc:1203] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:33.288748 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 44f747cd88e04d23951f351b1578bbb8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:33.269898 17723 raft_consensus.cc:348] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:33.308640 17723 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:33.308938 17723 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:33.294080 18024 tablet_copy_client.cc:308] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:33.310474 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet e6d3c011ad9546e3b69e23cbef587dbc from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:49117)
I20240502 08:18:33.310684 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet e6d3c011ad9546e3b69e23cbef587dbc from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:49117: session id = 42116b63c31a436ca060c34691780d04-e6d3c011ad9546e3b69e23cbef587dbc
I20240502 08:18:33.313777 14620 tablet_copy_source_session.cc:205] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:33.316790 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:33.319403 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d3c011ad9546e3b69e23cbef587dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.319211 14056 catalog_manager.cc:5484] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:33.324921 17658 tablet_bootstrap.cc:700] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Time spent opening tablet: real 0.110s	user 0.000s	sys 0.000s
I20240502 08:18:33.328261 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.332388 17989 ts_tablet_manager.cc:1875] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.335382 18024 tablet_copy_client.cc:783] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:33.335664 18024 tablet_copy_client.cc:647] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:33.339087 17723 ts_tablet_manager.cc:1398] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.079s	user 0.004s	sys 0.007s
I20240502 08:18:33.340023 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-76dcfec9be9f4f19a43d3f4603ae89d8 received from {username='slave'} at 127.6.151.72:49117
I20240502 08:18:33.340159 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 42116b63c31a436ca060c34691780d04-76dcfec9be9f4f19a43d3f4603ae89d8 on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:33.340757 17627 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:33.341074 17962 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:33.342576 14270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 103227us
I20240502 08:18:33.342903 18024 tablet_copy_client.cc:523] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:33.237363 17920 rpcz_store.cc:276] 0502 08:18:31.855545 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 08:18:31.855644 (+    99us) service_pool.cc:225] Handling call
0502 08:18:32.090259 (+234615us) ts_tablet_manager.cc:903] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
0502 08:18:32.694738 (+604479us) tablet_copy_client.cc:353] Tablet copy session begun
0502 08:18:32.828763 (+134025us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:32.828921 (+   158us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 08:18:33.209373 (+380452us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 08:18:33.209508 (+   135us) tablet_replica.cc:205] Creating consensus instance
0502 08:18:33.219871 (+ 10363us) ts_tablet_manager.cc:958] Replica registered.
0502 08:18:33.219902 (+    31us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":105,"tablet-copy.queue_time_us":234183,"thread_start_us":764531,"threads_started":5}
I20240502 08:18:33.343864 17920 tablet_copy_client.cc:783] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:33.344127 17920 tablet_copy_client.cc:647] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:33.345458 17948 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20240502 08:18:33.346061 17920 tablet_copy_client.cc:523] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:33.348834 18024 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:33.358007 14205 raft_consensus.cc:2932] T 44f747cd88e04d23951f351b1578bbb8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.367820 16532 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.372334 17995 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.382566 16533 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.403856 18063 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20240502 08:18:33.404417 14062 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.415809 16533 raft_consensus.cc:1203] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:33.420661 14206 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.424146 16533 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.424614 18063 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.432988 14056 catalog_manager.cc:5484] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.442457 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f747cd88e04d23951f351b1578bbb8 with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:33.442924 18055 tablet_replica.cc:330] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:33.443149 18055 raft_consensus.cc:2229] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.443387 18055 raft_consensus.cc:2258] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.443615 16902 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:33.444129 18055 ts_tablet_manager.cc:1875] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.451418 14058 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:33.461835 18055 ts_tablet_manager.cc:1888] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:33.464190 18055 log.cc:1196] T 44f747cd88e04d23951f351b1578bbb8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/44f747cd88e04d23951f351b1578bbb8
I20240502 08:18:33.469283 14334 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.472380 14596 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:33.475664 14469 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.496194 14205 consensus_queue.cc:237] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:33.498481 16135 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.499322 17995 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.500725 14598 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.502257 14203 consensus_queue.cc:237] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:33.503005 17995 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.506775 17920 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:33.506757 17991 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.510099 16268 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.511199 14598 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.515753 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet 44f747cd88e04d23951f351b1578bbb8 (table rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]) successfully deleted
I20240502 08:18:33.532440 18024 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:33.535627 16135 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.536032 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6379601e2b2d4bdabde2773fcbe06fb4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:33.539813 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.543936 17995 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.550067 17993 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.552547 14466 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.554611 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.556349 16269 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.560058 17993 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.561904 14466 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.572508 16135 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.579417 14334 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:33.598699 18024 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:33.600421 18024 ts_tablet_manager.cc:1367] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.252s	user 0.061s	sys 0.000s
I20240502 08:18:33.605584 18024 raft_consensus.cc:348] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:33.605898 18024 raft_consensus.cc:729] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:33.606115 18024 consensus_queue.cc:260] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:33.607311 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.613076 14334 consensus_queue.cc:237] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:33.615514 16533 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:33.615548 16135 raft_consensus.cc:1261] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:33.616236 18086 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:33.616995 18086 consensus_queue.cc:1035] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:33.620018 14061 catalog_manager.cc:5484] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.620240 18063 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.624995 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5341f4bf3c9643859859ed0d9887cfca with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:33.624598 16533 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.625687 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 784f72f28102445f8c4c44a3e3da0d2d with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:33.626127 18055 tablet_replica.cc:330] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:33.626338 18055 raft_consensus.cc:2229] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.626554 18055 raft_consensus.cc:2258] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.627739 18055 ts_tablet_manager.cc:1875] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.628547 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 784f72f28102445f8c4c44a3e3da0d2d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:33.632791 14062 catalog_manager.cc:5484] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.637141 14061 catalog_manager.cc:5484] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:33.639353 16135 raft_consensus.cc:2932] T 5341f4bf3c9643859859ed0d9887cfca P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:33.640377 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:33.643977 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 6379601e2b2d4bdabde2773fcbe06fb4 with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:33.652058 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:33.659121 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:33.670984 18024 ts_tablet_manager.cc:1398] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.070s	user 0.002s	sys 0.000s
I20240502 08:18:33.671168 16401 raft_consensus.cc:1203] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:33.691439 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.693710 18123 tablet_replica.cc:330] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:33.694063 18123 raft_consensus.cc:2229] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.695585 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-e6d3c011ad9546e3b69e23cbef587dbc received from {username='slave'} at 127.6.151.72:49117
I20240502 08:18:33.695721 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 42116b63c31a436ca060c34691780d04-e6d3c011ad9546e3b69e23cbef587dbc on tablet e6d3c011ad9546e3b69e23cbef587dbc with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:33.696954 18123 raft_consensus.cc:2258] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.708667 18123 ts_tablet_manager.cc:1875] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.717716 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.719288 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:33.724723 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:33.736886 14598 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:33.745112 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.746042 16268 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.748415 14468 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:33.749500 17697 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.753042 18055 ts_tablet_manager.cc:1888] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:33.760955 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 5341f4bf3c9643859859ed0d9887cfca with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:33.760937 17989 ts_tablet_manager.cc:1888] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:33.761448 17989 log.cc:1196] T 9ee2cdcfd53a469c803f846458dd083b P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ee2cdcfd53a469c803f846458dd083b
I20240502 08:18:33.770412 17993 raft_consensus.cc:1050] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:33.771359 17993 consensus_queue.cc:237] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:33.772729 17989 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53474 (request call id 27) took 1217 ms. Trace:
I20240502 08:18:33.772960 17989 rpcz_store.cc:276] 0502 08:18:32.544935 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 08:18:32.545336 (+   401us) service_pool.cc:225] Handling call
0502 08:18:33.135313 (+589977us) ts_tablet_manager.cc:1120] Deleting tablet 9ee2cdcfd53a469c803f846458dd083b
0502 08:18:33.135320 (+     7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:18:33.335799 (+200479us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:33.644581 (+308782us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:33.762005 (+117424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":589660}
I20240502 08:18:33.774441 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet 9ee2cdcfd53a469c803f846458dd083b (table rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]) successfully deleted
I20240502 08:18:33.774995 17989 tablet_replica.cc:330] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:33.777283 16533 raft_consensus.cc:1203] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:33.777344 17989 raft_consensus.cc:2229] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.778920 17989 raft_consensus.cc:2258] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.779670 18055 log.cc:1196] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/784f72f28102445f8c4c44a3e3da0d2d
I20240502 08:18:33.781152 18045 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.783073 17989 ts_tablet_manager.cc:1875] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.786322 17697 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:33.787945 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 930e3af93722422fb9237fd9b623ceac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:33.789024 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:33.790485 14062 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:33.798851 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 930e3af93722422fb9237fd9b623ceac with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:33.812397 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet 784f72f28102445f8c4c44a3e3da0d2d (table rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]) successfully deleted
I20240502 08:18:33.812994 18055 tablet_replica.cc:330] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:33.817538 14468 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:33.819890 16533 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:33.819893 16401 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:33.820566 16902 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:33.822934 17962 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:33.823989 17962 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.825559 16269 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.827342 16399 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.833045 18055 raft_consensus.cc:2229] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.836419 18123 ts_tablet_manager.cc:1888] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:33.836524 16533 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:33.837353 18123 log.cc:1196] T 6379601e2b2d4bdabde2773fcbe06fb4 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/6379601e2b2d4bdabde2773fcbe06fb4
I20240502 08:18:33.838932 18055 raft_consensus.cc:2258] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.840009 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet 6379601e2b2d4bdabde2773fcbe06fb4 (table rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]) successfully deleted
I20240502 08:18:33.840122 18127 ts_tablet_manager.cc:902] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:33.840744 18055 ts_tablet_manager.cc:1875] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.845110 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:33.846143 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6272b2f3a9504e01832cb31f5b322980 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:33.848697 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:33.848399 14056 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:33.867311 14468 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.870184 14466 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:33.872108 14334 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:33.872895 17993 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:33.873314 17995 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:33.875296 16268 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:33.875931 17995 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:33.912475 17989 ts_tablet_manager.cc:1888] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:33.916018 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 6272b2f3a9504e01832cb31f5b322980 with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:33.917042 18055 ts_tablet_manager.cc:1888] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:33.917181 18055 log.cc:1196] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/930e3af93722422fb9237fd9b623ceac
I20240502 08:18:33.917042 16269 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.918219 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet 930e3af93722422fb9237fd9b623ceac (table rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]) successfully deleted
I20240502 08:18:33.922063 14468 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.923440 14065 catalog_manager.cc:5484] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.915122 17382 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.939569 17989 log.cc:1196] T 5341f4bf3c9643859859ed0d9887cfca P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/5341f4bf3c9643859859ed0d9887cfca
I20240502 08:18:33.939632 14334 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:33.940366 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet 5341f4bf3c9643859859ed0d9887cfca (table rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]) successfully deleted
I20240502 08:18:33.940861 18127 tablet_copy_client.cc:308] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:33.941597 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 84eaabbe91524d129a170ad3519c265a from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:34071)
I20240502 08:18:33.941795 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 84eaabbe91524d129a170ad3519c265a from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:34071: session id = 5b84b46faddb47fcae86cc5f3f478bf1-84eaabbe91524d129a170ad3519c265a
I20240502 08:18:33.944988 14489 tablet_copy_source_session.cc:205] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:33.945675 18139 tablet_replica.cc:330] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:33.945910 18139 raft_consensus.cc:2229] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:33.949474 18045 raft_consensus.cc:1050] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:33.950328 18045 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:33.952893 18127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84eaabbe91524d129a170ad3519c265a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:33.960285 18139 raft_consensus.cc:2258] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:33.961962 18139 ts_tablet_manager.cc:1875] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:33.974310 14206 consensus_queue.cc:237] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:33.992131 18127 tablet_copy_client.cc:783] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:33.992444 18127 tablet_copy_client.cc:647] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:33.997635 18127 tablet_copy_client.cc:523] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:34.000550 16268 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.001272 17993 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.002431 18127 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:34.003646 14469 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.004953 16268 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.006850 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:34.012173 14056 catalog_manager.cc:5484] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.017652 16533 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.018394 18067 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.021037 18139 ts_tablet_manager.cc:1888] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:34.031123 18139 log.cc:1196] T 6272b2f3a9504e01832cb31f5b322980 P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/6272b2f3a9504e01832cb31f5b322980
I20240502 08:18:34.032373 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 6272b2f3a9504e01832cb31f5b322980 (table rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]) successfully deleted
I20240502 08:18:34.019037 17993 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.046584 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b15694b0b284ecebd899522dc69d422 with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:34.047086 18055 tablet_replica.cc:330] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:34.047367 18055 raft_consensus.cc:2229] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:34.047659 18055 raft_consensus.cc:2258] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:34.048094 17982 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20240502 08:18:34.049293 18055 ts_tablet_manager.cc:1875] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:34.050784 14206 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.052196 14469 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.054874 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b15694b0b284ecebd899522dc69d422 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:34.059183 18045 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.062171 14334 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.063043 18045 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.068403 18055 ts_tablet_manager.cc:1888] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:34.068773 18055 log.cc:1196] T 0b15694b0b284ecebd899522dc69d422 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b15694b0b284ecebd899522dc69d422
I20240502 08:18:34.081274 18067 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.091218 16533 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
W20240502 08:18:34.092007 17920 log.cc:924] Time spent T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20240502 08:18:34.100139 16902 raft_consensus.cc:1050] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:34.101241 16902 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:34.108327 14203 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.108590 14337 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.109259 17961 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.109627 17962 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.111819 14206 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.120522 17961 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.122325 14056 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.124727 16399 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.125463 17962 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.125483 14203 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.139505 14337 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.145038 17658 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:18:34.145643 17658 tablet_bootstrap.cc:492] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:34.145956 17658 ts_tablet_manager.cc:1367] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.940s	user 0.053s	sys 0.000s
I20240502 08:18:34.147153 17658 raft_consensus.cc:348] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:34.147923 17658 raft_consensus.cc:729] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:34.148141 17658 consensus_queue.cc:260] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:34.148802 17658 ts_tablet_manager.cc:1398] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:18:34.149507 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet 0b15694b0b284ecebd899522dc69d422 (table rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]) successfully deleted
I20240502 08:18:34.149991 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-1017d794c61345959c52f062dd39ef35 received from {username='slave'} at 127.6.151.71:52741
I20240502 08:18:34.150130 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-1017d794c61345959c52f062dd39ef35 on tablet 1017d794c61345959c52f062dd39ef35 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:34.146948 14334 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.158978 18127 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:34.159580 18127 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:34.159914 18127 ts_tablet_manager.cc:1367] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.158s	user 0.067s	sys 0.034s
I20240502 08:18:34.160830 18127 raft_consensus.cc:348] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:34.161070 18127 raft_consensus.cc:729] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:34.161250 18127 consensus_queue.cc:260] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:34.162382 18127 ts_tablet_manager.cc:1398] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 08:18:34.166729 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-84eaabbe91524d129a170ad3519c265a received from {username='slave'} at 127.6.151.70:34071
I20240502 08:18:34.166983 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-84eaabbe91524d129a170ad3519c265a on tablet 84eaabbe91524d129a170ad3519c265a with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:34.176843 14061 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:34.187450 16401 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.202977 17697 raft_consensus.cc:1050] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:34.203748 17697 consensus_queue.cc:237] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:34.266067 16533 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.266925 18067 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.277319 14203 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.278671 14334 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.279987 18067 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.280371 17697 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.297324 16268 raft_consensus.cc:1203] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:34.355423 18067 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.359972 14206 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.364327 16401 raft_consensus.cc:1203] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:34.364648 16401 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.373832 16532 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.374312 14065 catalog_manager.cc:5484] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.379251 14334 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.392786 14597 consensus_queue.cc:237] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:34.399755 16532 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.401608 17697 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.403132 14334 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.403853 17697 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.407284 17697 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.412412 14334 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.412876 16532 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.421056 14065 catalog_manager.cc:5484] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:34.422905 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6d3c011ad9546e3b69e23cbef587dbc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:34.426710 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d3c011ad9546e3b69e23cbef587dbc with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:34.427078 17989 tablet_replica.cc:330] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:34.427268 17989 raft_consensus.cc:2229] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:34.427529 17989 raft_consensus.cc:2258] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:34.431651 17989 ts_tablet_manager.cc:1875] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:34.462875 17989 ts_tablet_manager.cc:1888] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:34.463171 17989 log.cc:1196] T e6d3c011ad9546e3b69e23cbef587dbc P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/e6d3c011ad9546e3b69e23cbef587dbc
I20240502 08:18:34.464251 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet e6d3c011ad9546e3b69e23cbef587dbc (table rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]) successfully deleted
I20240502 08:18:34.467780 17719 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:34.468400 17920 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:34.469527 17719 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:34.472606 17719 ts_tablet_manager.cc:1367] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 1.607s	user 0.061s	sys 0.005s
I20240502 08:18:34.475150 17920 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:34.475792 17719 raft_consensus.cc:348] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:34.476011 17920 ts_tablet_manager.cc:1367] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.969s	user 0.066s	sys 0.000s
I20240502 08:18:34.476974 17920 raft_consensus.cc:348] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:34.477269 17920 raft_consensus.cc:729] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:34.477423 17719 raft_consensus.cc:729] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:34.477502 17920 consensus_queue.cc:260] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:34.477782 17719 consensus_queue.cc:260] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:34.483037 17719 ts_tablet_manager.cc:1398] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.001s
I20240502 08:18:34.491603 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-810c5f8a81ba4e0ab06129c0178f12e7 received from {username='slave'} at 127.6.151.72:49117
I20240502 08:18:34.491780 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 42116b63c31a436ca060c34691780d04-810c5f8a81ba4e0ab06129c0178f12e7 on tablet 810c5f8a81ba4e0ab06129c0178f12e7 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:34.518381 17920 ts_tablet_manager.cc:1398] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.042s	user 0.009s	sys 0.015s
I20240502 08:18:34.520409 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-7a92e7e903594d84a59f3ed058a5a3ec received from {username='slave'} at 127.6.151.69:40247
I20240502 08:18:34.520563 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-7a92e7e903594d84a59f3ed058a5a3ec on tablet 7a92e7e903594d84a59f3ed058a5a3ec with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:34.606410 14334 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:34.627029 14469 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:34.628592 14597 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:34.629952 14270 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97d7768da26e4fff966e494446a278f2. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:34.632438 17948 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:34.633119 17948 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:34.676481 17419 raft_consensus.cc:1050] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:34.677744 17419 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:34.703397 18063 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:34.707190 14596 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:34.720461 14065 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:34.728799 16135 raft_consensus.cc:1203] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:34.734242 16268 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.735421 17961 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.744397 16532 raft_consensus.cc:1203] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:34.747597 14597 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:34.765693 14334 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.766407 17419 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.766696 14596 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:34.767239 17419 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:34.774346 14469 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:34.780763 14596 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:34.783733 17982 raft_consensus.cc:1050] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:34.784628 16532 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.784806 17982 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:34.785517 14337 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.786419 18067 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.786621 14206 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.789758 18045 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.792449 18067 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.793747 17697 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.796351 14337 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.799866 16532 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.801851 14206 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.802137 14062 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:34.811005 16135 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:34.812803 18045 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:34.820312 14598 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:34.822623 17961 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.827373 14206 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:34.827953 18067 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:34.828333 14468 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:34.830789 17911 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.833946 14062 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:34.835007 14334 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.839088 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:34.840314 14203 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.847651 14058 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 6 to 7, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.848024 18045 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:34.848438 16268 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.850001 14598 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.852044 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:34.853039 16533 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:34.855630 17697 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:34.857244 16533 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.859918 18209 tablet_replica.cc:330] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:34.861474 14466 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:34.861958 18209 raft_consensus.cc:2229] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:34.862447 18209 raft_consensus.cc:2258] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:34.866678 18209 ts_tablet_manager.cc:1875] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:34.881808 14336 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.883024 17419 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.883361 18209 ts_tablet_manager.cc:1888] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:34.883612 18209 log.cc:1196] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/76dcfec9be9f4f19a43d3f4603ae89d8
I20240502 08:18:34.884631 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet 76dcfec9be9f4f19a43d3f4603ae89d8 (table rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]) successfully deleted
I20240502 08:18:34.888260 16401 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:34.888922 17995 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:34.888908 17419 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.892408 16267 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:34.893064 17995 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:34.893735 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84eaabbe91524d129a170ad3519c265a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:34.894234 14336 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.895726 14598 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:34.896304 17995 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:34.897037 14058 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:34.901094 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 84eaabbe91524d129a170ad3519c265a with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:34.909971 17697 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
W20240502 08:18:34.914705 14535 consensus_peers.cc:485] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d33a9164ef974a8caae83f3c22b87a51. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:34.915508 14535 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76dcfec9be9f4f19a43d3f4603ae89d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:34.920651 14065 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:34.922245 16135 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.929258 16269 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:34.930308 17961 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:34.931932 16269 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:34.956493 14468 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:34.963171 18219 tablet_replica.cc:330] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:34.963492 18219 raft_consensus.cc:2229] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:34.963855 18219 raft_consensus.cc:2258] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:34.971035 17982 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.971819 14597 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.978451 16268 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.980106 16401 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.976995 18219 ts_tablet_manager.cc:1875] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:34.985455 14062 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:34.998402 18219 ts_tablet_manager.cc:1888] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:35.004024 14468 consensus_queue.cc:237] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:35.015126 18219 log.cc:1196] T 84eaabbe91524d129a170ad3519c265a P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/84eaabbe91524d129a170ad3519c265a
I20240502 08:18:35.016721 14206 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:18:35.017283 14401 consensus_peers.cc:485] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fec3033140544f4994057d5b22973fff. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.017781 17961 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.020870 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 84eaabbe91524d129a170ad3519c265a (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:35.027033 14334 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.030712 17419 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.030546 16902 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.033466 14203 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.047874 14056 catalog_manager.cc:5484] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:35.054476 14597 consensus_queue.cc:237] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
W20240502 08:18:35.062590 14534 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88890b0913c0462bade8208dfa0f64b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.063648 18245 ts_tablet_manager.cc:902] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:35.064779 14334 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.065496 18067 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.072079 14336 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.073071 14206 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.075793 18067 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.096379 18067 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:35.099047 14206 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:35.099454 14334 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:35.107451 14062 catalog_manager.cc:5484] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:35.110836 17993 raft_consensus.cc:1050] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:35.113454 17993 consensus_queue.cc:237] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:35.134474 16135 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.136725 14597 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.137790 14468 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.139581 17995 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:35.140393 17995 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:35.144382 17993 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:35.219446 17995 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:35.228441 16138 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:35.228835 14336 consensus_queue.cc:237] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:35.236557 14065 catalog_manager.cc:5484] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:35.238929 18245 tablet_copy_client.cc:308] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:35.239665 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 97d7768da26e4fff966e494446a278f2 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:57815)
I20240502 08:18:35.239869 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 97d7768da26e4fff966e494446a278f2 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:57815: session id = 42116b63c31a436ca060c34691780d04-97d7768da26e4fff966e494446a278f2
I20240502 08:18:35.249367 14203 consensus_queue.cc:237] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:35.253268 14469 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:35.255234 17993 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:35.256750 14469 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:35.259116 16137 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:35.260082 17982 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:35.267576 14357 tablet_copy_source_session.cc:205] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:35.269413 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d7768da26e4fff966e494446a278f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:35.276504 18245 tablet_copy_client.cc:783] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:35.276851 18245 tablet_copy_client.cc:647] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:35.282217 18245 tablet_copy_client.cc:523] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:35.287199 18245 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:35.287576 16137 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:35.291599 14466 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:35.298060 14059 catalog_manager.cc:5484] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
W20240502 08:18:35.300455 14270 consensus_peers.cc:485] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e901787c37a40e9a7c6db44091d11ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.304270 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a92e7e903594d84a59f3ed058a5a3ec with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:35.304733 18219 tablet_replica.cc:330] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:35.305009 18219 raft_consensus.cc:2229] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:35.305227 18219 pending_rounds.cc:62] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Trying to abort 1 pending ops.
I20240502 08:18:35.305348 18219 pending_rounds.cc:68] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023156900295319552 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7a92e7e903594d84a59f3ed058a5a3ec" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } } }
I20240502 08:18:35.306000 18219 raft_consensus.cc:2894] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 08:18:35.306134 18219 raft_consensus.cc:2866] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:35.306294 18219 raft_consensus.cc:2258] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:35.285254 18227 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:35.319960 14203 consensus_queue.cc:237] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:35.326516 14597 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.328588 14469 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.330462 18157 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.331115 14336 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:35.331619 18157 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:18:35.334772 14270 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.371214 18045 raft_consensus.cc:1050] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:35.372201 18045 consensus_queue.cc:237] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:35.389144 14598 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.390326 14466 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.400386 18157 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:35.400898 18157 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:35.405365 18157 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.413352 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a92e7e903594d84a59f3ed058a5a3ec with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:35.418556 18245 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:35.419126 18245 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:35.419545 18245 ts_tablet_manager.cc:1367] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.132s	user 0.056s	sys 0.007s
I20240502 08:18:35.420564 18245 raft_consensus.cc:348] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:35.420935 18245 raft_consensus.cc:729] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:35.421186 18245 consensus_queue.cc:260] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:35.428157 18219 ts_tablet_manager.cc:1875] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:35.430915 18245 ts_tablet_manager.cc:1398] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.011s	user 0.000s	sys 0.008s
I20240502 08:18:35.437887 18219 ts_tablet_manager.cc:1888] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:35.438058 18219 log.cc:1196] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/7a92e7e903594d84a59f3ed058a5a3ec
I20240502 08:18:35.441588 18290 ts_tablet_manager.cc:902] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:35.444047 18292 ts_tablet_manager.cc:902] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:35.446382 16532 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.447640 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 7a92e7e903594d84a59f3ed058a5a3ec (table rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]) successfully deleted
I20240502 08:18:35.447583 18045 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:35.448653 14466 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.450165 14597 raft_consensus.cc:1261] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.450682 18227 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.463683 14062 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.469085 14596 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.477023 14466 raft_consensus.cc:1261] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.477769 18227 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.478310 14469 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.492159 18045 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:35.495316 14141 consensus_peers.cc:485] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 827a2fb0396441749a35977e2854bfb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.495716 17995 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:35.505484 18063 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:35.505901 14065 catalog_manager.cc:5484] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:35.508204 14469 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:35.532083 14065 catalog_manager.cc:5484] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:35.538704 14596 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:35.557911 18290 tablet_copy_client.cc:308] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:35.564853 18292 tablet_copy_client.cc:308] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:35.570830 17911 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:35.579070 14469 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:35.580300 14056 catalog_manager.cc:5484] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:35.581456 16533 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
W20240502 08:18:35.596602 14272 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 102199us
I20240502 08:18:35.598198 14598 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:35.598393 14596 consensus_queue.cc:237] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:35.598539 14468 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:35.615289 16532 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:35.615979 18045 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:35.616235 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-97d7768da26e4fff966e494446a278f2 received from {username='slave'} at 127.6.151.72:57815
I20240502 08:18:35.616353 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session 42116b63c31a436ca060c34691780d04-97d7768da26e4fff966e494446a278f2 on tablet 97d7768da26e4fff966e494446a278f2 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:35.621098 18294 ts_tablet_manager.cc:902] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:35.628615 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:35.634924 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:35.639568 17911 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:35.644009 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 810c5f8a81ba4e0ab06129c0178f12e7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:35.646719 16533 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:35.646706 14065 catalog_manager.cc:5484] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:35.657415 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:35.663503 14336 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:35.667358 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 810c5f8a81ba4e0ab06129c0178f12e7 with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:35.688580 18342 tablet_replica.cc:330] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:35.690928 18245 ts_tablet_manager.cc:902] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:35.692983 14466 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:35.694003 17911 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:35.696161 14466 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:35.699839 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:35.707938 18341 ts_tablet_manager.cc:902] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:35.718997 18342 raft_consensus.cc:2229] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:35.719553 18342 raft_consensus.cc:2258] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:35.720597 18342 ts_tablet_manager.cc:1875] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:35.689970 14271 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97d7768da26e4fff966e494446a278f2. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.731222 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:35.746495 18294 tablet_copy_client.cc:308] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:35.747579 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet fec3033140544f4994057d5b22973fff from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:44057)
I20240502 08:18:35.747810 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet fec3033140544f4994057d5b22973fff from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:44057: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-fec3033140544f4994057d5b22973fff
I20240502 08:18:35.755290 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:40315)
I20240502 08:18:35.755506 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:40315: session id = b3dfd9d6472342689afd86203f7afe5f-76dcfec9be9f4f19a43d3f4603ae89d8
I20240502 08:18:35.757143 14489 tablet_copy_source_session.cc:205] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:35.759117 14620 tablet_copy_source_session.cc:205] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:35.761752 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet d33a9164ef974a8caae83f3c22b87a51 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:40315)
I20240502 08:18:35.762027 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet d33a9164ef974a8caae83f3c22b87a51 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:40315: session id = b3dfd9d6472342689afd86203f7afe5f-d33a9164ef974a8caae83f3c22b87a51
I20240502 08:18:35.765173 14620 tablet_copy_source_session.cc:205] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:35.766726 18290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dcfec9be9f4f19a43d3f4603ae89d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:35.766960 18292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a9164ef974a8caae83f3c22b87a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:35.774514 18290 tablet_copy_client.cc:783] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:35.774953 18290 tablet_copy_client.cc:647] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:35.776425 18294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec3033140544f4994057d5b22973fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:35.795279 18292 tablet_copy_client.cc:783] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:35.795585 18292 tablet_copy_client.cc:647] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:35.800226 18290 tablet_copy_client.cc:523] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:35.814400 16532 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:35.824347 18292 tablet_copy_client.cc:523] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:35.835268 14598 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
W20240502 08:18:35.841261 14535 consensus_peers.cc:485] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.842033 16137 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:35.842052 14468 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:35.842728 18045 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:35.843528 18067 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:35.849344 18067 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:35.856951 16137 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:35.857905 14065 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:35.860005 14336 consensus_queue.cc:237] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:35.862748 18292 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:35.870383 14469 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:35.880606 18341 tablet_copy_client.cc:308] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:35.883525 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 88890b0913c0462bade8208dfa0f64b7 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:38521)
I20240502 08:18:35.883790 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 88890b0913c0462bade8208dfa0f64b7 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:38521: session id = 5b84b46faddb47fcae86cc5f3f478bf1-88890b0913c0462bade8208dfa0f64b7
I20240502 08:18:35.888630 14620 tablet_copy_source_session.cc:205] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:35.892443 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88890b0913c0462bade8208dfa0f64b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:35.894773 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:35.901628 18290 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:35.909647 18294 tablet_copy_client.cc:783] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:35.916103 14203 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.916857 18308 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.925225 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:35.927842 18342 ts_tablet_manager.cc:1888] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:35.930840 18294 tablet_copy_client.cc:647] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:35.930990 14598 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:35.944052 18294 tablet_copy_client.cc:523] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 08:18:35.945719 14533 consensus_peers.cc:485] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d33a9164ef974a8caae83f3c22b87a51. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:35.932219 18308 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:35.978610 18342 log.cc:1196] T 810c5f8a81ba4e0ab06129c0178f12e7 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/810c5f8a81ba4e0ab06129c0178f12e7
I20240502 08:18:35.979910 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet 810c5f8a81ba4e0ab06129c0178f12e7 (table rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]) successfully deleted
I20240502 08:18:35.991739 18157 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:35.995673 14206 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:35.999192 14596 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:36.004619 14059 catalog_manager.cc:5484] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index -1 to 4, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:36.011904 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:36.012773 18294 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:36.014878 18377 ts_tablet_manager.cc:902] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:36.044775 18381 ts_tablet_manager.cc:902] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
W20240502 08:18:36.054510 14271 consensus_peers.cc:485] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0815524d0eb448438dae05f646f14580. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.057586 18341 tablet_copy_client.cc:783] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:36.057969 18341 tablet_copy_client.cc:647] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:36.059805 18245 tablet_copy_client.cc:308] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:36.061041 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 6e901787c37a40e9a7c6db44091d11ef from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:57815)
I20240502 08:18:36.061218 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 6e901787c37a40e9a7c6db44091d11ef from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:57815: session id = 42116b63c31a436ca060c34691780d04-6e901787c37a40e9a7c6db44091d11ef
I20240502 08:18:36.064296 14357 tablet_copy_source_session.cc:205] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:36.065678 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e901787c37a40e9a7c6db44091d11ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:36.066155 18381 tablet_copy_client.cc:308] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:36.073796 18341 tablet_copy_client.cc:523] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:36.075915 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 97d7768da26e4fff966e494446a278f2 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:39627)
I20240502 08:18:36.076200 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 97d7768da26e4fff966e494446a278f2 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:39627: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-97d7768da26e4fff966e494446a278f2
I20240502 08:18:36.079232 14357 tablet_copy_source_session.cc:205] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:36.103368 18341 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:36.106961 18245 tablet_copy_client.cc:783] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:36.111815 18245 tablet_copy_client.cc:647] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:36.114141 18245 tablet_copy_client.cc:523] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:36.115701 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d7768da26e4fff966e494446a278f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:36.122071 18245 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:36.161128 18377 tablet_copy_client.cc:308] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:36.162045 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 827a2fb0396441749a35977e2854bfb6 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:52741)
I20240502 08:18:36.162256 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 827a2fb0396441749a35977e2854bfb6 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:52741: session id = b3dfd9d6472342689afd86203f7afe5f-827a2fb0396441749a35977e2854bfb6
I20240502 08:18:36.165328 14226 tablet_copy_source_session.cc:205] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:36.169046 18377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827a2fb0396441749a35977e2854bfb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:36.197930 18292 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:36.200511 18381 tablet_copy_client.cc:783] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:36.200804 18381 tablet_copy_client.cc:647] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:36.211355 18381 tablet_copy_client.cc:523] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:36.222237 18377 tablet_copy_client.cc:783] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:36.223711 18292 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:36.224018 18292 ts_tablet_manager.cc:1367] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.361s	user 0.045s	sys 0.013s
I20240502 08:18:36.224840 18292 raft_consensus.cc:348] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:36.225100 18292 raft_consensus.cc:729] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:36.225297 18292 consensus_queue.cc:260] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:36.229138 18377 tablet_copy_client.cc:647] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:36.234704 18377 tablet_copy_client.cc:523] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:36.236390 14469 consensus_queue.cc:237] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:36.238276 18377 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:36.263809 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:36.267861 18290 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:36.268357 18290 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:36.268627 18290 ts_tablet_manager.cc:1367] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.367s	user 0.043s	sys 0.012s
I20240502 08:18:36.269341 18290 raft_consensus.cc:348] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:36.269619 18290 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:36.269801 18290 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:36.278815 18290 ts_tablet_manager.cc:1398] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.000s
I20240502 08:18:36.279706 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-76dcfec9be9f4f19a43d3f4603ae89d8 received from {username='slave'} at 127.6.151.71:40315
I20240502 08:18:36.279868 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-76dcfec9be9f4f19a43d3f4603ae89d8 on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:36.288058 18381 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:36.294533 18308 raft_consensus.cc:1050] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:36.295750 18308 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
W20240502 08:18:36.299237 14402 consensus_peers.cc:485] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fec3033140544f4994057d5b22973fff. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.299937 14468 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:36.300184 14598 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:36.300634 14336 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:36.301859 18284 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:36.301971 14206 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:36.302349 18284 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:36.302691 17961 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:18:36.302919 14271 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:36.303195 14401 consensus_peers.cc:485] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:36.314347 17961 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:36.315917 18245 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:36.315838 18284 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:36.316450 18245 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:36.316720 18245 ts_tablet_manager.cc:1367] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.195s	user 0.051s	sys 0.001s
I20240502 08:18:36.317553 18245 raft_consensus.cc:348] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:36.317824 18245 raft_consensus.cc:729] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:36.318045 18245 consensus_queue.cc:260] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:36.324656 14596 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:36.326097 18292 ts_tablet_manager.cc:1398] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.102s	user 0.001s	sys 0.024s
I20240502 08:18:36.328336 18245 ts_tablet_manager.cc:1398] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.001s
I20240502 08:18:36.328509 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-d33a9164ef974a8caae83f3c22b87a51 received from {username='slave'} at 127.6.151.71:40315
I20240502 08:18:36.328346 16532 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:36.329587 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-6e901787c37a40e9a7c6db44091d11ef received from {username='slave'} at 127.6.151.72:57815
I20240502 08:18:36.329696 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session 42116b63c31a436ca060c34691780d04-6e901787c37a40e9a7c6db44091d11ef on tablet 6e901787c37a40e9a7c6db44091d11ef with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:36.330168 18063 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:36.332554 16532 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:36.334822 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-d33a9164ef974a8caae83f3c22b87a51 on tablet d33a9164ef974a8caae83f3c22b87a51 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:36.343484 14059 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:36.345649 14468 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:36.350952 18377 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:36.351464 18377 tablet_bootstrap.cc:492] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:36.351763 18377 ts_tablet_manager.cc:1367] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.114s	user 0.049s	sys 0.001s
I20240502 08:18:36.352607 18377 raft_consensus.cc:348] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:36.352870 18377 raft_consensus.cc:729] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:36.353099 18377 consensus_queue.cc:260] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:36.353703 18377 ts_tablet_manager.cc:1398] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 08:18:36.354472 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-827a2fb0396441749a35977e2854bfb6 received from {username='slave'} at 127.6.151.71:52741
I20240502 08:18:36.354657 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-827a2fb0396441749a35977e2854bfb6 on tablet 827a2fb0396441749a35977e2854bfb6 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:36.377866 17961 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:36.378926 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:36.382371 14206 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:36.388340 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:36.391979 14061 catalog_manager.cc:5484] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:36.398743 18341 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:36.399260 18341 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:36.398885 14336 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:36.399551 18341 ts_tablet_manager.cc:1367] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.296s	user 0.051s	sys 0.006s
I20240502 08:18:36.400534 18341 raft_consensus.cc:348] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:36.400872 18341 raft_consensus.cc:729] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:36.401082 18341 consensus_queue.cc:260] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:36.426437 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:36.438670 18341 ts_tablet_manager.cc:1398] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.039s	user 0.018s	sys 0.019s
I20240502 08:18:36.439517 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-88890b0913c0462bade8208dfa0f64b7 received from {username='slave'} at 127.6.151.70:38521
I20240502 08:18:36.439710 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-88890b0913c0462bade8208dfa0f64b7 on tablet 88890b0913c0462bade8208dfa0f64b7 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:36.448453 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:36.461252 18245 ts_tablet_manager.cc:902] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:36.475364 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:36.487473 18341 ts_tablet_manager.cc:902] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:36.493647 16401 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:36.509943 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:36.534327 16401 raft_consensus.cc:1203] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:36.534399 18341 tablet_copy_client.cc:308] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:36.541002 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:36.582350 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 0815524d0eb448438dae05f646f14580 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:42663)
I20240502 08:18:36.582569 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 0815524d0eb448438dae05f646f14580 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:42663: session id = 5b84b46faddb47fcae86cc5f3f478bf1-0815524d0eb448438dae05f646f14580
I20240502 08:18:36.588582 16401 raft_consensus.cc:1203] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:36.589542 14357 tablet_copy_source_session.cc:205] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:36.591051 18341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0815524d0eb448438dae05f646f14580. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:36.596343 18341 tablet_copy_client.cc:783] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:36.596709 18341 tablet_copy_client.cc:647] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:36.598567 18341 tablet_copy_client.cc:523] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:36.604194 18341 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:36.632474 16267 raft_consensus.cc:1203] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:36.664080 18245 tablet_copy_client.cc:308] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:36.665041 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet d33a9164ef974a8caae83f3c22b87a51 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:49117)
I20240502 08:18:36.665259 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet d33a9164ef974a8caae83f3c22b87a51 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:49117: session id = 42116b63c31a436ca060c34691780d04-d33a9164ef974a8caae83f3c22b87a51
I20240502 08:18:36.669114 14620 tablet_copy_source_session.cc:205] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:36.670785 18245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a9164ef974a8caae83f3c22b87a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:36.709023 18341 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:36.709496 18341 tablet_bootstrap.cc:492] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:36.709753 18341 ts_tablet_manager.cc:1367] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.106s	user 0.068s	sys 0.019s
I20240502 08:18:36.710508 18341 raft_consensus.cc:348] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:36.710865 18341 raft_consensus.cc:729] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:36.711167 18341 consensus_queue.cc:260] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:36.712047 18341 ts_tablet_manager.cc:1398] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 08:18:36.712869 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-0815524d0eb448438dae05f646f14580 received from {username='slave'} at 127.6.151.70:42663
I20240502 08:18:36.713047 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-0815524d0eb448438dae05f646f14580 on tablet 0815524d0eb448438dae05f646f14580 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:36.723361 16533 raft_consensus.cc:1203] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:36.750734 18245 tablet_copy_client.cc:783] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:36.751300 18245 tablet_copy_client.cc:647] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:36.755335 18245 tablet_copy_client.cc:523] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:36.793619 18245 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:36.813731 14336 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:36.817175 14598 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:36.818215 14468 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:36.819478 16532 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:36.820600 18308 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:36.821079 18308 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:36.821739 18284 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:36.828492 18157 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:36.829493 14468 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:36.831414 14598 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:36.833447 16533 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:36.836392 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97d7768da26e4fff966e494446a278f2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:36.839367 14065 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:36.844820 16117 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d7768da26e4fff966e494446a278f2 with delete_type TABLET_DATA_TOMBSTONED (TS f28bbf93da0c45eaaae0959e7a2f1e7e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45260
W20240502 08:18:36.852394 14050 catalog_manager.cc:4810] TS f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): delete failed for tablet 97d7768da26e4fff966e494446a278f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97d7768da26e4fff966e494446a278f2 already in progress: copying tablet
I20240502 08:18:36.873235 16267 raft_consensus.cc:1203] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:36.907387 18478 ts_tablet_manager.cc:902] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:36.951488 17995 raft_consensus.cc:1050] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:36.952702 17995 consensus_queue.cc:237] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:36.967156 18483 raft_consensus.cc:1050] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:36.968037 18483 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:36.970963 18478 tablet_copy_client.cc:308] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:36.971879 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet fec3033140544f4994057d5b22973fff from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:59591)
I20240502 08:18:36.972085 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet fec3033140544f4994057d5b22973fff from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:59591: session id = b3dfd9d6472342689afd86203f7afe5f-fec3033140544f4994057d5b22973fff
I20240502 08:18:36.976004 14468 raft_consensus.cc:1261] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:36.976912 18447 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:36.977241 14598 raft_consensus.cc:1261] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:36.978163 16401 raft_consensus.cc:1261] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:36.980279 17993 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:36.981364 17993 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:36.996034 17993 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:36.999204 18294 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:36.999680 18294 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:36.999989 18294 ts_tablet_manager.cc:1367] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.987s	user 0.053s	sys 0.008s
I20240502 08:18:37.003309 18294 raft_consensus.cc:348] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:37.003576 18294 raft_consensus.cc:729] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:37.003790 18294 consensus_queue.cc:260] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:37.013005 18045 raft_consensus.cc:1050] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:37.013962 18045 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:37.016985 18294 ts_tablet_manager.cc:1398] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.002s
I20240502 08:18:37.018184 14488 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-fec3033140544f4994057d5b22973fff received from {username='slave'} at 127.6.151.69:44057
I20240502 08:18:37.018352 14488 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-fec3033140544f4994057d5b22973fff on tablet fec3033140544f4994057d5b22973fff with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:37.021804 16401 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.026844 14489 tablet_copy_source_session.cc:205] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:37.028867 18478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fec3033140544f4994057d5b22973fff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:37.029562 16533 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:37.030575 14206 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:37.031251 18067 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:37.032477 16400 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:37.035912 18478 tablet_copy_client.cc:783] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:37.035889 18067 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:37.036162 18478 tablet_copy_client.cc:647] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:37.036489 18067 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:37.039867 14469 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.040318 14062 catalog_manager.cc:5484] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 4 to 5, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:37.054315 14203 consensus_queue.cc:237] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:37.068030 16137 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:18:37.068689 14533 consensus_peers.cc:485] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.069500 14466 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:37.069793 18483 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:37.071663 18067 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:37.074942 16399 raft_consensus.cc:1261] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.079411 14468 raft_consensus.cc:1261] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.081910 18447 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.082331 16400 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:37.082501 18447 consensus_queue.cc:1035] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.083513 18067 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:37.087021 17911 raft_consensus.cc:1050] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:37.087857 17911 consensus_queue.cc:237] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:37.092957 18381 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.093433 18381 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:37.093406 17995 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.095448 14468 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.100168 18067 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:37.100848 16135 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:37.105484 18381 ts_tablet_manager.cc:1367] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.818s	user 0.066s	sys 0.005s
I20240502 08:18:37.106477 18381 raft_consensus.cc:348] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:37.106817 18381 raft_consensus.cc:729] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:37.107026 18381 consensus_queue.cc:260] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:37.108561 18478 tablet_copy_client.cc:523] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:37.109730 18381 ts_tablet_manager.cc:1398] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:18:37.110522 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-97d7768da26e4fff966e494446a278f2 received from {username='slave'} at 127.6.151.69:39627
I20240502 08:18:37.110563 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 827a2fb0396441749a35977e2854bfb6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:37.110723 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-97d7768da26e4fff966e494446a278f2 on tablet 97d7768da26e4fff966e494446a278f2 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:37.113441 18478 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:37.113494 16401 raft_consensus.cc:2932] T 827a2fb0396441749a35977e2854bfb6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.114585 14062 catalog_manager.cc:5484] T 827a2fb0396441749a35977e2854bfb6 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:37.118156 14065 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:37.130625 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 827a2fb0396441749a35977e2854bfb6 with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:37.132887 16117 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d7768da26e4fff966e494446a278f2 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:45260
I20240502 08:18:37.133249 18473 tablet_replica.cc:330] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: stopping tablet replica
I20240502 08:18:37.133435 18473 raft_consensus.cc:2229] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:18:37.133621 18473 raft_consensus.cc:2258] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:18:37.134320 18473 ts_tablet_manager.cc:1875] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.127215 18524 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.142437 16399 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.142377 16533 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.143128 18473 ts_tablet_manager.cc:1888] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:37.143352 18473 log.cc:1196] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/wals/97d7768da26e4fff966e494446a278f2
I20240502 08:18:37.144232 14050 catalog_manager.cc:4830] TS f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): tablet 97d7768da26e4fff966e494446a278f2 (table rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]) successfully deleted
I20240502 08:18:37.145794 16400 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:37.148250 14065 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.148250 14206 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.152309 14466 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:37.171013 18245 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:37.171641 18245 tablet_bootstrap.cc:492] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:37.173314 18245 ts_tablet_manager.cc:1367] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.380s	user 0.055s	sys 0.010s
I20240502 08:18:37.174394 18245 raft_consensus.cc:348] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:37.176512 18245 raft_consensus.cc:729] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:37.176788 18245 consensus_queue.cc:260] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:37.177438 18245 ts_tablet_manager.cc:1398] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 08:18:37.178382 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-d33a9164ef974a8caae83f3c22b87a51 received from {username='slave'} at 127.6.151.72:49117
I20240502 08:18:37.178547 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 42116b63c31a436ca060c34691780d04-d33a9164ef974a8caae83f3c22b87a51 on tablet d33a9164ef974a8caae83f3c22b87a51 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:37.182060 16267 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.183527 14336 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.183545 14206 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.184231 18045 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.184811 18067 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.187846 18525 tablet_replica.cc:330] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:37.188203 18525 raft_consensus.cc:2229] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:37.188589 18525 pending_rounds.cc:62] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Trying to abort 1 pending ops.
I20240502 08:18:37.188733 18525 pending_rounds.cc:68] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023156907835383808 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "827a2fb0396441749a35977e2854bfb6" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } } }
I20240502 08:18:37.189452 18525 raft_consensus.cc:2894] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 08:18:37.189652 18525 raft_consensus.cc:2866] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:37.189841 18525 raft_consensus.cc:2258] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:37.190654 17911 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.197698 14598 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:37.201432 18063 raft_consensus.cc:1050] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:37.202263 18063 consensus_queue.cc:237] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:37.207840 16533 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.208869 18287 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.211956 14598 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.212698 14468 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.213129 18287 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.213433 18308 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.214459 16137 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:37.215322 18483 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:37.216158 14469 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:37.216985 18045 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:37.218896 16400 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:37.220352 18523 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:37.229959 18067 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.232820 18525 ts_tablet_manager.cc:1875] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.233016 16400 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.236104 18524 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.240108 14336 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.241384 14056 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.244325 14469 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.250658 14065 catalog_manager.cc:5484] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:37.257786 16267 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.265058 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:37.265686 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d33a9164ef974a8caae83f3c22b87a51 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:37.276723 14596 consensus_queue.cc:237] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:37.279053 16267 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.279760 18523 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.283183 18524 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.284420 18287 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:37.285224 14469 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:37.287209 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88890b0913c0462bade8208dfa0f64b7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:37.287607 14466 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:37.293670 14598 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:37.296252 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:37.298851 16513 tablet_service.cc:1514] Processing DeleteTablet for tablet d33a9164ef974a8caae83f3c22b87a51 with delete_type TABLET_DATA_TOMBSTONED (TS 42116b63c31a436ca060c34691780d04 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34882
I20240502 08:18:37.305862 16267 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.314117 14065 catalog_manager.cc:5484] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:37.314834 16137 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.317327 16135 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.317358 16533 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:37.318060 17961 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.327060 14336 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.327801 17911 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.329666 14334 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.331634 14206 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.332485 17961 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.335152 14598 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.335763 17961 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.335769 14059 catalog_manager.cc:5484] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 5 to 6, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.343529 18556 tablet_replica.cc:330] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: stopping tablet replica
I20240502 08:18:37.343986 18556 raft_consensus.cc:2229] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:18:37.344281 18556 raft_consensus.cc:2258] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:18:37.344566 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 88890b0913c0462bade8208dfa0f64b7 with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:37.350081 18556 ts_tablet_manager.cc:1875] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.362097 14334 consensus_queue.cc:237] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:37.365514 14598 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.366393 18063 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.369194 18287 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:37.369828 14598 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:37.373840 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e901787c37a40e9a7c6db44091d11ef with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:37.374922 16533 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.378247 16135 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.379312 14056 catalog_manager.cc:5484] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:37.382699 14598 consensus_queue.cc:237] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:37.384009 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e901787c37a40e9a7c6db44091d11ef with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:37.385840 16400 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:37.385927 16135 raft_consensus.cc:1261] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:37.386427 17911 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:37.389279 17911 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.393923 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d33a9164ef974a8caae83f3c22b87a51 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:37.394284 16135 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.399281 18566 tablet_replica.cc:330] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:37.399638 18566 raft_consensus.cc:2229] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:37.399943 18566 pending_rounds.cc:62] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Trying to abort 1 pending ops.
W20240502 08:18:37.399928 14051 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet d33a9164ef974a8caae83f3c22b87a51 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER failed with leader e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 10 but the committed config has opid_index of 11
I20240502 08:18:37.400961 17961 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.404067 18063 raft_consensus.cc:1050] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:37.404920 18063 consensus_queue.cc:237] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:37.407312 16135 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.410995 14062 catalog_manager.cc:5484] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 10 to 11, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:37.416565 14059 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:37.422149 14203 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.400221 18566 pending_rounds.cc:68] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023156908389609472 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "88890b0913c0462bade8208dfa0f64b7" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } } }
I20240502 08:18:37.424551 18566 raft_consensus.cc:2894] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 08:18:37.424749 18566 raft_consensus.cc:2866] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:37.424923 18566 raft_consensus.cc:2258] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:37.428696 14598 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.436343 14447 tablet_service.cc:1514] Processing DeleteTablet for tablet d33a9164ef974a8caae83f3c22b87a51 with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:37.437913 18308 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.440579 14469 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:37.442215 16533 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:37.446120 16135 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:37.447336 18576 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:37.453748 18566 ts_tablet_manager.cc:1875] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.457880 18525 ts_tablet_manager.cc:1888] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:37.458513 18525 log.cc:1196] T 827a2fb0396441749a35977e2854bfb6 P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/827a2fb0396441749a35977e2854bfb6
I20240502 08:18:37.464973 18569 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.469466 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f109afd932d4c77a6b7cd63c417bc70 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:37.471529 16135 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.484439 14056 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.486374 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:37.492846 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 827a2fb0396441749a35977e2854bfb6 (table rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]) successfully deleted
I20240502 08:18:37.493435 14206 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:37.498261 14469 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:37.500310 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f109afd932d4c77a6b7cd63c417bc70 with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:37.504773 18525 tablet_replica.cc:330] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:37.505182 18525 raft_consensus.cc:2229] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:37.505527 18525 raft_consensus.cc:2258] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:37.507050 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:37.509511 18067 consensus_queue.cc:1035] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:37.510982 16400 raft_consensus.cc:2932] T d33a9164ef974a8caae83f3c22b87a51 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:37.516907 14596 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.517087 16267 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.517786 18157 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.518404 18157 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.523500 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:37.526419 16137 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:37.534353 14206 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:37.539789 14206 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:37.540599 18583 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20240502 08:18:37.541890 18566 ts_tablet_manager.cc:1888] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
W20240502 08:18:37.542728 14402 consensus_peers.cc:485] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f109afd932d4c77a6b7cd63c417bc70. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.543114 18576 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:37.543689 17961 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.544715 18556 ts_tablet_manager.cc:1888] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:37.544859 18556 log.cc:1196] T d33a9164ef974a8caae83f3c22b87a51 P 42116b63c31a436ca060c34691780d04: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/wals/d33a9164ef974a8caae83f3c22b87a51
I20240502 08:18:37.546877 18287 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.547583 16137 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.548820 14051 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4f109afd932d4c77a6b7cd63c417bc70 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:37.552202 14065 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 7 to 8, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:37.543622 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:37.545305 14206 raft_consensus.cc:858] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:37.557318 18157 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
W20240502 08:18:37.557684 14206 raft_consensus.cc:1565] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 8' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }: 
  New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:37.558342 14206 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:37.561122 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:37.563714 14596 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.567972 18566 log.cc:1196] T 88890b0913c0462bade8208dfa0f64b7 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/88890b0913c0462bade8208dfa0f64b7
I20240502 08:18:37.569029 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet 88890b0913c0462bade8208dfa0f64b7 (table rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]) successfully deleted
I20240502 08:18:37.569856 14058 catalog_manager.cc:5484] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.595760 18525 ts_tablet_manager.cc:1875] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.595595 16267 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.596302 14206 consensus_queue.cc:237] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:37.601039 14205 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.604847 18525 ts_tablet_manager.cc:1888] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:18:37.608598 16267 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:37.609262 14140 consensus_peers.cc:485] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd647b6897bc4292a2dc70ef39abab5e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.609454 17995 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:37.611387 18570 tablet_replica.cc:330] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:37.611652 18570 raft_consensus.cc:2229] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:37.613590 14469 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:37.614204 17995 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:37.620666 18570 raft_consensus.cc:2258] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:37.622056 18570 ts_tablet_manager.cc:1875] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.628460 17995 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:37.631429 16268 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:37.651743 14062 catalog_manager.cc:5484] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:37.668532 14469 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:37.690799 18525 log.cc:1196] T 4f109afd932d4c77a6b7cd63c417bc70 P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/4f109afd932d4c77a6b7cd63c417bc70
I20240502 08:18:37.691480 14334 consensus_queue.cc:237] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:37.691816 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 4f109afd932d4c77a6b7cd63c417bc70 (table rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]) successfully deleted
I20240502 08:18:37.694759 16268 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.695701 14206 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.696498 18287 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.697198 18287 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:37.698369 14206 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.702159 18157 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.706684 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0815524d0eb448438dae05f646f14580 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:37.707733 14049 catalog_manager.cc:4830] TS 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): tablet d33a9164ef974a8caae83f3c22b87a51 (table rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]) successfully deleted
I20240502 08:18:37.709654 16268 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.710388 14061 catalog_manager.cc:5484] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:37.714812 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 0815524d0eb448438dae05f646f14580 with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:37.716885 18525 tablet_replica.cc:330] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:37.717123 18525 raft_consensus.cc:2229] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:37.717593 18525 raft_consensus.cc:2258] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:37.719583 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:37.724270 18525 ts_tablet_manager.cc:1875] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.757862 17961 raft_consensus.cc:1050] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:37.758778 18525 ts_tablet_manager.cc:1888] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:37.759167 18525 log.cc:1196] T 0815524d0eb448438dae05f646f14580 P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/0815524d0eb448438dae05f646f14580
I20240502 08:18:37.759125 17961 consensus_queue.cc:237] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
W20240502 08:18:37.762382 14402 consensus_peers.cc:485] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:37.763587 14205 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.764308 18565 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.765658 14334 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.766515 18565 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.760391 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 0815524d0eb448438dae05f646f14580 (table rebalance_test_table_2 [id=1f43ee8167df4e8dbbdcb01012aec36c]) successfully deleted
I20240502 08:18:37.784886 18583 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.790234 14336 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.797345 14206 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.798519 14203 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:37.827461 16137 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:37.829739 18583 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:37.832203 14065 catalog_manager.cc:5484] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 5 to 6, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:37.832942 16135 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:37.883728 18570 ts_tablet_manager.cc:1888] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:37.883998 18570 log.cc:1196] T 6e901787c37a40e9a7c6db44091d11ef P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/6e901787c37a40e9a7c6db44091d11ef
I20240502 08:18:37.885013 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet 6e901787c37a40e9a7c6db44091d11ef (table rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]) successfully deleted
I20240502 08:18:37.885299 18570 tablet_replica.cc:330] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:37.885600 18570 raft_consensus.cc:2229] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:37.886252 18570 raft_consensus.cc:2258] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:37.910183 18570 ts_tablet_manager.cc:1875] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:37.959744 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:37.961731 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:37.982189 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:38.001626 18570 ts_tablet_manager.cc:1888] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 08:18:38.012224 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:38.021338 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.021850 14206 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:38.035280 18636 ts_tablet_manager.cc:902] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:38.045360 16533 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:38.045969 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.047040 14334 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:38.048100 17993 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:38.049885 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:38.051297 17995 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:38.063086 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:38.103176 18656 ts_tablet_manager.cc:902] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:38.129343 14469 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:38.133191 17993 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:38.159477 18570 log.cc:1196] T d33a9164ef974a8caae83f3c22b87a51 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/d33a9164ef974a8caae83f3c22b87a51
I20240502 08:18:38.162035 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet d33a9164ef974a8caae83f3c22b87a51 (table rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]) successfully deleted
I20240502 08:18:38.172713 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.175735 18656 tablet_copy_client.cc:308] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:38.177115 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet cd647b6897bc4292a2dc70ef39abab5e from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:40247)
I20240502 08:18:38.177327 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet cd647b6897bc4292a2dc70ef39abab5e from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:40247: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-cd647b6897bc4292a2dc70ef39abab5e
I20240502 08:18:38.177371 18636 tablet_copy_client.cc:308] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:38.178267 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 4f109afd932d4c77a6b7cd63c417bc70 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:59591)
I20240502 08:18:38.178434 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 4f109afd932d4c77a6b7cd63c417bc70 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:59591: session id = b3dfd9d6472342689afd86203f7afe5f-4f109afd932d4c77a6b7cd63c417bc70
I20240502 08:18:38.180178 14226 tablet_copy_source_session.cc:205] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.182933 14489 tablet_copy_source_session.cc:205] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:38.187515 18656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd647b6897bc4292a2dc70ef39abab5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.189247 18636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f109afd932d4c77a6b7cd63c417bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:38.196302 17995 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:38.199023 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.200250 16532 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:38.203409 18656 tablet_copy_client.cc:783] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.219569 14065 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:38.226497 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:38.238006 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:38.240633 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:38.246361 18656 tablet_copy_client.cc:647] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.257511 14206 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:38.269941 18656 tablet_copy_client.cc:523] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.275287 18656 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
W20240502 08:18:38.281407 14140 consensus_peers.cc:485] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0bcdf713f124d46a22ddfc2873cd742. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.283658 14334 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:38.300050 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:38.302948 14469 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.304560 18447 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:38.309249 16533 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.310075 18447 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:38.311262 14466 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:38.320485 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.330708 18447 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:38.334544 14336 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.336199 14469 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:38.342814 14049 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f0bcdf713f124d46a22ddfc2873cd742 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:38.342809 17993 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:38.344282 14062 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:38.355787 16532 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:38.365890 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:38.369777 14334 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:38.407474 14466 consensus_queue.cc:237] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:38.410889 14334 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.411777 16135 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.412667 18565 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:38.415393 18583 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:38.422081 18565 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:38.424592 14334 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:38.440186 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fec3033140544f4994057d5b22973fff with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:38.441613 16135 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:38.448599 14062 catalog_manager.cc:5484] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:38.456315 18478 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.457526 18478 tablet_bootstrap.cc:492] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:38.458686 18478 ts_tablet_manager.cc:1367] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 1.345s	user 0.050s	sys 0.011s
I20240502 08:18:38.462662 18636 tablet_copy_client.cc:783] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:38.463158 18636 tablet_copy_client.cc:647] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:38.480574 14206 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.494560 14206 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
W20240502 08:18:38.498493 14140 consensus_peers.cc:485] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0bcdf713f124d46a22ddfc2873cd742. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.499663 16532 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:38.500377 17995 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:38.502933 17995 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:38.507819 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0bcdf713f124d46a22ddfc2873cd742 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:38.510218 16532 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:38.512255 14062 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:38.516659 14334 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:38.527144 17995 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:38.529439 14400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 47491.3us
I20240502 08:18:38.529057 14336 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
W20240502 08:18:38.530529 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.66:44269, user_credentials={real_user=slave}} blocked reactor thread for 60260.9us
I20240502 08:18:38.531848 18478 raft_consensus.cc:348] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:38.536902 18636 tablet_copy_client.cc:523] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:38.542685 18478 raft_consensus.cc:729] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:38.543064 18478 consensus_queue.cc:260] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:38.544338 18636 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:38.556931 16382 tablet_service.cc:1514] Processing DeleteTablet for tablet fec3033140544f4994057d5b22973fff with delete_type TABLET_DATA_TOMBSTONED (TS b3dfd9d6472342689afd86203f7afe5f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53026
W20240502 08:18:38.575767 14051 catalog_manager.cc:4810] TS b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): delete failed for tablet fec3033140544f4994057d5b22973fff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fec3033140544f4994057d5b22973fff already in progress: copying tablet
I20240502 08:18:38.527057 18692 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20240502 08:18:38.603739 18478 ts_tablet_manager.cc:1398] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.145s	user 0.001s	sys 0.004s
I20240502 08:18:38.608167 14203 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:38.608865 16382 tablet_service.cc:1514] Processing DeleteTablet for tablet fec3033140544f4994057d5b22973fff with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:53026
I20240502 08:18:38.609257 18704 tablet_replica.cc:330] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: stopping tablet replica
I20240502 08:18:38.609798 18704 raft_consensus.cc:2229] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:18:38.610122 18704 raft_consensus.cc:2258] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:18:38.611150 18704 ts_tablet_manager.cc:1875] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:38.617916 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bcdf713f124d46a22ddfc2873cd742 with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:38.620893 18570 tablet_replica.cc:330] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:38.620985 18704 ts_tablet_manager.cc:1888] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:18:38.621155 18704 log.cc:1196] T fec3033140544f4994057d5b22973fff P b3dfd9d6472342689afd86203f7afe5f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal/wals/fec3033140544f4994057d5b22973fff
I20240502 08:18:38.621490 18656 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.621924 18656 tablet_bootstrap.cc:492] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:38.621995 14051 catalog_manager.cc:4830] TS b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): tablet fec3033140544f4994057d5b22973fff (table rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]) successfully deleted
I20240502 08:18:38.622205 18656 ts_tablet_manager.cc:1367] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.347s	user 0.051s	sys 0.004s
I20240502 08:18:38.623070 18656 raft_consensus.cc:348] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:38.623355 18656 raft_consensus.cc:729] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:38.623557 18656 consensus_queue.cc:260] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:38.629248 18656 ts_tablet_manager.cc:1398] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20240502 08:18:38.633615 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-cd647b6897bc4292a2dc70ef39abab5e received from {username='slave'} at 127.6.151.69:40247
I20240502 08:18:38.633805 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-cd647b6897bc4292a2dc70ef39abab5e on tablet cd647b6897bc4292a2dc70ef39abab5e with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:38.649035 18570 raft_consensus.cc:2229] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:38.650810 18570 raft_consensus.cc:2258] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:38.652108 18570 ts_tablet_manager.cc:1875] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:38.664907 14400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 43545.6us
I20240502 08:18:38.674526 18570 ts_tablet_manager.cc:1888] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:18:38.674741 18570 log.cc:1196] T f0bcdf713f124d46a22ddfc2873cd742 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/f0bcdf713f124d46a22ddfc2873cd742
I20240502 08:18:38.675729 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet f0bcdf713f124d46a22ddfc2873cd742 (table rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]) successfully deleted
I20240502 08:18:38.676735 14466 consensus_queue.cc:237] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:38.679947 14336 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:38.682541 18698 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:38.685844 18698 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:38.690999 14336 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
W20240502 08:18:38.702744 14401 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 128431us
I20240502 08:18:38.708515 16135 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:38.712397 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-fec3033140544f4994057d5b22973fff received from {username='slave'} at 127.6.151.71:59591
I20240502 08:18:38.712572 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-fec3033140544f4994057d5b22973fff on tablet fec3033140544f4994057d5b22973fff with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:38.714993 14469 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:38.717609 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fec3033140544f4994057d5b22973fff with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:38.724433 14062 catalog_manager.cc:5484] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 7 to 8, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:38.725032 14336 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } }
I20240502 08:18:38.729775 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet fec3033140544f4994057d5b22973fff with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:38.735714 18709 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:38.737991 16135 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:38.741091 18714 tablet_replica.cc:330] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:38.741322 18714 raft_consensus.cc:2229] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:38.743937 18714 raft_consensus.cc:2258] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:38.747838 16400 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:38.748471 16532 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 08:18:38.748471 14272 consensus_peers.cc:485] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): Couldn't send request to peer e79a5862de19490b89a9ea97860d776b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.748680 18063 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:38.749207 18287 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:38.756295 18714 ts_tablet_manager.cc:1875] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:38.768330 16531 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:38.771584 16399 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:38.772235 14596 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:38.773293 18709 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:38.773875 18709 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:38.782486 14061 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 9 to 10, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:38.798859 18583 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:38.801081 18287 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:38.804041 14336 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:38.806049 16399 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:38.813263 14061 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:38.822044 16400 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:38.824841 18709 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:38.827454 14596 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:38.838855 14469 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
W20240502 08:18:38.842406 14401 consensus_peers.cc:485] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a635dc6d64724295be4b4a7f68d0fad6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:38.842950 16400 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.846660 18706 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:38.852932 14598 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.854014 18583 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:38.857968 14598 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:38.867079 14061 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:38.857211 18706 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:38.871407 18714 ts_tablet_manager.cc:1888] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:18:38.871647 18714 log.cc:1196] T fec3033140544f4994057d5b22973fff P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/fec3033140544f4994057d5b22973fff
I20240502 08:18:38.872835 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet fec3033140544f4994057d5b22973fff (table rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]) successfully deleted
I20240502 08:18:38.874538 18636 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:38.875468 14051 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a635dc6d64724295be4b4a7f68d0fad6 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:38.876286 16400 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:38.876279 18636 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:38.882884 18636 ts_tablet_manager.cc:1367] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.339s	user 0.061s	sys 0.000s
I20240502 08:18:38.883771 18636 raft_consensus.cc:348] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:38.888116 18636 raft_consensus.cc:729] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:38.888427 18636 consensus_queue.cc:260] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:38.892354 18636 ts_tablet_manager.cc:1398] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.000s
I20240502 08:18:38.893972 14336 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:38.896510 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-4f109afd932d4c77a6b7cd63c417bc70 received from {username='slave'} at 127.6.151.71:59591
I20240502 08:18:38.896661 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-4f109afd932d4c77a6b7cd63c417bc70 on tablet 4f109afd932d4c77a6b7cd63c417bc70 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:38.907621 14334 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:38.908597 18696 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:38.910413 14334 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:38.969401 16135 raft_consensus.cc:1203] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:38.979825 18747 ts_tablet_manager.cc:902] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:39.016494 18747 tablet_copy_client.cc:308] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:39.017686 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet f0bcdf713f124d46a22ddfc2873cd742 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:50837)
I20240502 08:18:39.017875 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet f0bcdf713f124d46a22ddfc2873cd742 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:50837: session id = 5b84b46faddb47fcae86cc5f3f478bf1-f0bcdf713f124d46a22ddfc2873cd742
I20240502 08:18:39.018071 14598 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:39.020812 14226 tablet_copy_source_session.cc:205] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:39.022562 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0bcdf713f124d46a22ddfc2873cd742. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:39.025015 14596 consensus_queue.cc:237] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:39.030755 14336 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:39.031383 16531 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:39.031478 14534 consensus_peers.cc:485] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6d3c011ad9546e3b69e23cbef587dbc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:39.032200 18632 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:39.034584 18632 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:39.039327 14203 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.039593 18747 tablet_copy_client.cc:783] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:39.039953 18747 tablet_copy_client.cc:647] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:39.044155 16400 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.046099 16532 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.046464 18632 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.048293 18633 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.048933 18633 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.050122 18747 tablet_copy_client.cc:523] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:39.065547 18747 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:39.119673 16533 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.125244 18632 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.145680 16400 raft_consensus.cc:1203] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:18:39.151854 14061 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.158417 16400 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.161815 14598 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
W20240502 08:18:39.165668 14534 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76dcfec9be9f4f19a43d3f4603ae89d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:39.167868 16399 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:39.168712 18633 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:39.173933 16533 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:39.174973 18632 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:39.189020 18633 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.194492 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:39.195703 14334 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.226704 14065 catalog_manager.cc:5484] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.235967 18791 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.237073 16533 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.245754 18747 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:39.246176 14056 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 9 to 10, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.252596 18747 tablet_bootstrap.cc:492] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:39.253051 18747 ts_tablet_manager.cc:1367] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.188s	user 0.061s	sys 0.008s
I20240502 08:18:39.254202 18747 raft_consensus.cc:348] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:39.255501 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:39.256275 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:39.257480 16531 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.254576 18747 raft_consensus.cc:729] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:39.262900 18747 consensus_queue.cc:260] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:39.270980 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:39.308674 14203 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:39.309432 18632 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.310940 14203 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.320510 18800 ts_tablet_manager.cc:902] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:39.345316 18800 tablet_copy_client.cc:308] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:39.346226 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet cf7124419bae40dfaaabdd2e673a3eb0 from peer e79a5862de19490b89a9ea97860d776b ({username='slave'} at 127.6.151.68:36491)
I20240502 08:18:39.346419 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet cf7124419bae40dfaaabdd2e673a3eb0 from peer e79a5862de19490b89a9ea97860d776b at {username='slave'} at 127.6.151.68:36491: session id = e79a5862de19490b89a9ea97860d776b-cf7124419bae40dfaaabdd2e673a3eb0
I20240502 08:18:39.350984 14357 tablet_copy_source_session.cc:205] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:39.353621 18800 ts_tablet_manager.cc:1875] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:39.369472 18683 raft_consensus.cc:1050] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:39.370406 18683 consensus_queue.cc:237] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:39.372042 18800 ts_tablet_manager.cc:1888] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 08:18:39.377030 14469 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:39.377101 16137 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:39.377593 18800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7124419bae40dfaaabdd2e673a3eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:39.377846 16268 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:39.377962 18683 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:39.378758 18447 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:39.379313 18447 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:39.387073 18800 tablet_copy_client.cc:783] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:39.386916 14466 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:39.387491 18800 tablet_copy_client.cc:647] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:39.390925 18800 tablet_copy_client.cc:523] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 08:18:39.390951 14401 consensus_peers.cc:485] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a635dc6d64724295be4b4a7f68d0fad6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:39.392879 14334 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:39.393088 16400 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:39.393785 18583 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:39.394217 18583 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:39.403272 18800 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:39.403517 18583 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.410465 14334 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.418195 14056 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.422827 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a635dc6d64724295be4b4a7f68d0fad6 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:39.433661 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet a635dc6d64724295be4b4a7f68d0fad6 with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:39.447835 16399 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.450109 18811 tablet_replica.cc:330] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:39.450342 18811 raft_consensus.cc:2229] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:39.450745 18811 raft_consensus.cc:2258] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:39.451836 18811 ts_tablet_manager.cc:1875] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:39.481604 14203 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.484551 18811 ts_tablet_manager.cc:1888] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:18:39.485610 18811 log.cc:1196] T a635dc6d64724295be4b4a7f68d0fad6 P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/a635dc6d64724295be4b4a7f68d0fad6
I20240502 08:18:39.486861 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet a635dc6d64724295be4b4a7f68d0fad6 (table rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]) successfully deleted
I20240502 08:18:39.499068 18747 ts_tablet_manager.cc:1398] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.246s	user 0.029s	sys 0.065s
I20240502 08:18:39.501215 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-f0bcdf713f124d46a22ddfc2873cd742 received from {username='slave'} at 127.6.151.70:50837
I20240502 08:18:39.501405 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-f0bcdf713f124d46a22ddfc2873cd742 on tablet f0bcdf713f124d46a22ddfc2873cd742 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:39.502890 14051 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:39.503094 18747 ts_tablet_manager.cc:902] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:39.557564 18800 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:39.558076 18800 tablet_bootstrap.cc:492] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Bootstrap complete.
I20240502 08:18:39.558382 18800 ts_tablet_manager.cc:1367] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.155s	user 0.067s	sys 0.004s
I20240502 08:18:39.561152 18800 raft_consensus.cc:348] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } }
I20240502 08:18:39.561471 18800 raft_consensus.cc:729] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: LEARNER
I20240502 08:18:39.561679 18800 consensus_queue.cc:260] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } }
I20240502 08:18:39.562302 18800 ts_tablet_manager.cc:1398] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:18:39.563256 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session e79a5862de19490b89a9ea97860d776b-cf7124419bae40dfaaabdd2e673a3eb0 received from {username='slave'} at 127.6.151.68:36491
I20240502 08:18:39.563436 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session e79a5862de19490b89a9ea97860d776b-cf7124419bae40dfaaabdd2e673a3eb0 on tablet cf7124419bae40dfaaabdd2e673a3eb0 with peer e79a5862de19490b89a9ea97860d776b
I20240502 08:18:39.579815 18698 raft_consensus.cc:1050] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:39.580772 18698 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:39.601724 18806 ts_tablet_manager.cc:902] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:39.612717 18683 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.613605 16269 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.619802 14469 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.621440 14056 catalog_manager.cc:5484] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.626493 16400 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.671497 14206 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:39.683811 18813 ts_tablet_manager.cc:902] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:39.686409 14205 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.687072 16135 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.687695 18583 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.690519 18698 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.706075 16138 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.721506 16400 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.722219 18698 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.737059 18747 tablet_copy_client.cc:308] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:39.751358 14336 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.751622 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet e6d3c011ad9546e3b69e23cbef587dbc from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:38521)
I20240502 08:18:39.751827 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet e6d3c011ad9546e3b69e23cbef587dbc from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:38521: session id = 5b84b46faddb47fcae86cc5f3f478bf1-e6d3c011ad9546e3b69e23cbef587dbc
I20240502 08:18:39.753957 18583 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.755502 14620 tablet_copy_source_session.cc:205] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:39.756028 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:39.758558 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:39.758668 16268 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.759508 17995 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.759537 16400 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.761365 17995 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:18:39.763222 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:39.763903 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d3c011ad9546e3b69e23cbef587dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:39.765812 17995 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.767459 14056 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:39.768014 14205 consensus_queue.cc:237] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:39.773066 16267 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.775139 18747 tablet_copy_client.cc:783] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:39.775408 18747 tablet_copy_client.cc:647] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:39.778023 14061 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:39.789121 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:39.792575 14203 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.796375 16138 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.796552 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:39.798996 18747 tablet_copy_client.cc:523] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:39.812753 18747 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:39.812747 14334 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.815452 17995 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.820982 16135 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:39.840224 18683 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.855011 14598 raft_consensus.cc:1203] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 08:18:39.857506 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd647b6897bc4292a2dc70ef39abab5e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:39.875319 14336 consensus_queue.cc:237] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:39.878747 14598 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:39.883846 16531 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:39.884418 16399 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:39.885495 18633 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:39.888450 14469 raft_consensus.cc:1261] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:39.888428 18633 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.889016 18287 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:18:39.889607 16267 raft_consensus.cc:1261] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:39.890388 17995 consensus_queue.cc:1035] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:39.891539 16399 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.892078 16268 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.895388 18786 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:39.898749 14598 raft_consensus.cc:1261] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:18:39.899724 16531 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.900815 18851 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 08:18:39.904179 18287 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.910591 14466 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.914945 16134 raft_consensus.cc:2932] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
W20240502 08:18:39.921648 14271 consensus_peers.cc:485] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52331b69903943cb9d74f5ef5787dd00. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:39.932251 16268 raft_consensus.cc:1203] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
W20240502 08:18:39.936082 14534 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76dcfec9be9f4f19a43d3f4603ae89d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:39.939253 18747 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:39.939698 18806 tablet_copy_client.cc:308] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:39.939754 18747 tablet_bootstrap.cc:492] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:39.940023 18747 ts_tablet_manager.cc:1367] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.127s	user 0.049s	sys 0.006s
I20240502 08:18:39.940587 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet a635dc6d64724295be4b4a7f68d0fad6 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:44057)
I20240502 08:18:39.940796 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet a635dc6d64724295be4b4a7f68d0fad6 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:44057: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-a635dc6d64724295be4b4a7f68d0fad6
I20240502 08:18:39.943861 14489 tablet_copy_source_session.cc:205] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:39.945405 18747 raft_consensus.cc:348] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:39.945688 18747 raft_consensus.cc:729] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:39.946040 18747 consensus_queue.cc:260] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:39.946988 18747 ts_tablet_manager.cc:1398] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.001s
I20240502 08:18:39.947885 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-e6d3c011ad9546e3b69e23cbef587dbc received from {username='slave'} at 127.6.151.70:38521
I20240502 08:18:39.948048 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-e6d3c011ad9546e3b69e23cbef587dbc on tablet e6d3c011ad9546e3b69e23cbef587dbc with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:39.950445 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:39.951514 14058 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 10 to 11, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:39.952612 14056 catalog_manager.cc:5484] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:39.953229 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a635dc6d64724295be4b4a7f68d0fad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:39.961364 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:39.971940 18869 tablet_replica.cc:330] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:39.974490 18813 tablet_copy_client.cc:308] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
W20240502 08:18:39.974917 14272 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.67:39913, user_credentials={real_user=slave}} blocked reactor thread for 59476.2us
I20240502 08:18:39.975772 14598 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:39.974210 14058 catalog_manager.cc:5484] T cd647b6897bc4292a2dc70ef39abab5e P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:39.982436 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:38521)
I20240502 08:18:39.982610 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:38521: session id = 5b84b46faddb47fcae86cc5f3f478bf1-76dcfec9be9f4f19a43d3f4603ae89d8
I20240502 08:18:39.985554 14620 tablet_copy_source_session.cc:205] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:39.993882 18806 tablet_copy_client.cc:783] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:39.994187 18806 tablet_copy_client.cc:647] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:39.996193 18806 tablet_copy_client.cc:523] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:39.999676 18806 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:40.000344 18869 raft_consensus.cc:2229] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:40.003360 18869 raft_consensus.cc:2258] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:40.004608 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dcfec9be9f4f19a43d3f4603ae89d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:40.004623 18869 ts_tablet_manager.cc:1875] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:40.023613 18869 ts_tablet_manager.cc:1888] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 08:18:40.028163 18869 log.cc:1196] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/76dcfec9be9f4f19a43d3f4603ae89d8
I20240502 08:18:40.034560 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet 76dcfec9be9f4f19a43d3f4603ae89d8 (table rebalance_test_table_8 [id=3ebcd209cc5e4e148d4fb46dfb1008af]) successfully deleted
I20240502 08:18:40.037916 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet cd647b6897bc4292a2dc70ef39abab5e with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:40.069276 18880 tablet_replica.cc:330] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:40.069658 18880 raft_consensus.cc:2229] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:40.070062 18880 raft_consensus.cc:2258] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:40.078182 18880 ts_tablet_manager.cc:1875] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:40.097564 18806 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:40.098054 18806 tablet_bootstrap.cc:492] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:40.098340 18806 ts_tablet_manager.cc:1367] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.099s	user 0.056s	sys 0.001s
I20240502 08:18:40.099120 18806 raft_consensus.cc:348] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:40.099375 18806 raft_consensus.cc:729] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:40.099556 18806 consensus_queue.cc:260] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:40.100322 14206 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:40.100878 18806 ts_tablet_manager.cc:1398] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 08:18:40.102694 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-a635dc6d64724295be4b4a7f68d0fad6 received from {username='slave'} at 127.6.151.69:44057
I20240502 08:18:40.102826 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-a635dc6d64724295be4b4a7f68d0fad6 on tablet a635dc6d64724295be4b4a7f68d0fad6 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:40.104543 18813 tablet_copy_client.cc:783] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
W20240502 08:18:40.105952 14139 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c1fecce883346459e21c29b98d95b7b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:40.104841 18813 tablet_copy_client.cc:647] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
W20240502 08:18:40.107321 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:40.108155 18813 tablet_copy_client.cc:523] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:40.108528 14336 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:40.109205 18683 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:40.111783 18813 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:40.111920 16268 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:40.112687 18683 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:40.115121 18447 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:40.118520 18880 ts_tablet_manager.cc:1888] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:40.120043 14336 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:40.121594 18880 log.cc:1196] T cd647b6897bc4292a2dc70ef39abab5e P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/cd647b6897bc4292a2dc70ef39abab5e
I20240502 08:18:40.122576 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet cd647b6897bc4292a2dc70ef39abab5e (table rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]) successfully deleted
I20240502 08:18:40.128111 16268 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:40.139914 14062 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 9 to 10, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:40.217943 14206 consensus_queue.cc:237] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:40.226969 16135 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:18:40.227074 14141 consensus_peers.cc:485] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6379601e2b2d4bdabde2773fcbe06fb4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:40.227910 18683 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:40.230914 14598 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.231602 18447 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:40.236461 18813 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:40.237084 18813 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:40.237457 18813 ts_tablet_manager.cc:1367] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.126s	user 0.070s	sys 0.014s
I20240502 08:18:40.239069 18813 raft_consensus.cc:348] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:40.239356 18813 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:40.239604 18813 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:40.240433 18813 ts_tablet_manager.cc:1398] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
I20240502 08:18:40.241544 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-76dcfec9be9f4f19a43d3f4603ae89d8 received from {username='slave'} at 127.6.151.70:38521
I20240502 08:18:40.241681 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-76dcfec9be9f4f19a43d3f4603ae89d8 on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:40.244138 18447 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:40.248580 14598 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:40.249732 16138 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:40.253931 16135 raft_consensus.cc:1203] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:18:40.255882 14058 catalog_manager.cc:5484] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:40.274833 16268 raft_consensus.cc:1203] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:40.369951 14469 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:40.379807 16399 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:40.380471 18698 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:40.388192 14206 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:40.388856 18698 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:40.391938 18861 raft_consensus.cc:1050] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER e79a5862de19490b89a9ea97860d776b to VOTER
I20240502 08:18:40.392869 18861 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } }
I20240502 08:18:40.393671 17995 raft_consensus.cc:1050] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:40.394084 14336 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:40.394536 17995 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:40.395381 18806 ts_tablet_manager.cc:902] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:40.395825 18698 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:40.408097 16531 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:40.408695 14598 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:40.409130 16401 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:40.409951 18287 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:40.410097 16268 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.410784 18447 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.410583 18287 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20240502 08:18:40.412173 18862 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:40.418149 16532 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.419119 14334 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.420506 18447 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.421324 18447 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.422417 14337 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:40.432090 16533 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.433612 18698 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:40.436031 14469 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.437373 14596 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.443794 14336 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:40.447309 18723 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:40.449038 18723 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:40.456166 18287 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:40.473555 18287 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:40.475536 16531 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:40.480398 17995 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.491019 18813 tablet_replica.cc:330] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: stopping tablet replica
I20240502 08:18:40.491632 18813 raft_consensus.cc:2229] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:18:40.491923 18813 raft_consensus.cc:2258] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:18:40.491808 18723 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.493444 18813 ts_tablet_manager.cc:1875] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:40.497025 16532 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.501250 14334 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.510129 18813 ts_tablet_manager.cc:1888] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 08:18:40.510351 18813 log.cc:1196] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/wals/76dcfec9be9f4f19a43d3f4603ae89d8
I20240502 08:18:40.510924 18813 ts_tablet_manager.cc:902] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
W20240502 08:18:40.522316 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.71:39897, user_credentials={real_user=slave}} blocked reactor thread for 79410.6us
I20240502 08:18:40.526093 18806 tablet_copy_client.cc:308] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:40.527307 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 52331b69903943cb9d74f5ef5787dd00 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:39627)
I20240502 08:18:40.527508 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 52331b69903943cb9d74f5ef5787dd00 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:39627: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-52331b69903943cb9d74f5ef5787dd00
I20240502 08:18:40.529413 16401 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:40.532245 14206 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:40.539928 14058 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 5 to 6. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:40.554226 14469 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:40.556281 16533 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.559648 14357 tablet_copy_source_session.cc:205] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:40.561372 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52331b69903943cb9d74f5ef5787dd00. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:40.566545 14401 consensus_peers.cc:485] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b44cc3d7fc9944c7abf33b4da92b2f6f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:40.567528 18806 tablet_copy_client.cc:783] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:40.567767 18806 tablet_copy_client.cc:647] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:40.570835 18806 tablet_copy_client.cc:523] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:40.579341 18806 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:40.628279 14203 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.629544 18698 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:18:40.630607 14141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.67:39913, user_credentials={real_user=slave}} blocked reactor thread for 99722.8us
W20240502 08:18:40.635000 14140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:45703, user_credentials={real_user=slave}} blocked reactor thread for 111190us
I20240502 08:18:40.639346 18990 ts_tablet_manager.cc:902] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:40.654434 16268 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.654620 14598 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.660533 16399 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.661188 14062 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 10 to 11, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.661540 18698 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:40.662149 18990 tablet_copy_client.cc:308] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:40.663378 14065 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.664467 18706 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:40.666968 14337 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.667671 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 6379601e2b2d4bdabde2773fcbe06fb4 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:52741)
I20240502 08:18:40.667737 18698 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:40.667869 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 6379601e2b2d4bdabde2773fcbe06fb4 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:52741: session id = b3dfd9d6472342689afd86203f7afe5f-6379601e2b2d4bdabde2773fcbe06fb4
I20240502 08:18:40.669410 14337 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:40.672757 16399 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:40.673038 14058 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 7 to 8. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:40.677289 14226 tablet_copy_source_session.cc:205] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:40.678365 14051 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b44cc3d7fc9944c7abf33b4da92b2f6f with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:40.682571 18791 raft_consensus.cc:1050] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:40.683666 18791 consensus_queue.cc:237] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:40.686820 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6379601e2b2d4bdabde2773fcbe06fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:40.700515 14469 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:40.700749 14057 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:40.710630 18813 tablet_copy_client.cc:308] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:40.715970 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:38521)
I20240502 08:18:40.716178 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:38521: session id = 5b84b46faddb47fcae86cc5f3f478bf1-76dcfec9be9f4f19a43d3f4603ae89d8
I20240502 08:18:40.720669 14620 tablet_copy_source_session.cc:205] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:40.723690 18990 tablet_copy_client.cc:783] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:40.728744 18990 tablet_copy_client.cc:647] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:40.698810 16401 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.731276 18813 ts_tablet_manager.cc:1875] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:40.731499 18990 tablet_copy_client.cc:523] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:40.731573 14336 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
W20240502 08:18:40.734135 14271 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:40.734696 16531 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.735553 18976 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.735862 14336 consensus_queue.cc:237] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } }
I20240502 08:18:40.743938 18990 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:40.734196 14596 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:40.745440 19005 ts_tablet_manager.cc:902] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:40.752779 14469 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.755016 14597 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240502 08:18:40.757349 14051 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 97d7768da26e4fff966e494446a278f2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER failed with leader 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }.
  Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:40.758462 18287 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:40.759546 18287 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.767553 16401 raft_consensus.cc:1261] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:40.768492 19007 ts_tablet_manager.cc:902] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:40.771055 16268 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.771772 18786 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:40.779788 16533 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.780799 18786 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:40.784806 14336 consensus_queue.cc:237] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:40.789006 14334 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.789573 18633 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:40.792817 14469 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:40.807554 18706 raft_consensus.cc:1050] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:40.808702 18706 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:40.809247 18287 consensus_queue.cc:1035] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:40.810573 18977 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
W20240502 08:18:40.816009 14401 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84eaabbe91524d129a170ad3519c265a. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:40.815292 14400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 135852us
I20240502 08:18:40.816287 18862 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.819454 16267 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.820084 14206 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:40.820504 18698 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:40.820681 14469 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:40.825564 16531 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:40.826444 16138 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.827314 18698 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.829283 14062 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:40.836570 16401 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.836577 14598 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.837560 14335 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.837659 18583 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.839810 18698 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.842330 16533 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.843735 18956 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:40.846563 14598 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.846109 14466 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:40.847620 14598 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.847563 14065 catalog_manager.cc:5484] T cf7124419bae40dfaaabdd2e673a3eb0 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:40.853637 14336 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:40.854254 18698 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:40.854605 14051 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 97d7768da26e4fff966e494446a278f2 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:40.854967 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf7124419bae40dfaaabdd2e673a3eb0 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:40.855562 16513 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7124419bae40dfaaabdd2e673a3eb0 with delete_type TABLET_DATA_TOMBSTONED (TS 42116b63c31a436ca060c34691780d04 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:34882
I20240502 08:18:40.869171 18633 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.869681 16269 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.876235 16531 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.877130 18583 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:40.879735 16267 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:40.878535 14334 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.886993 14062 catalog_manager.cc:5484] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 7 to 8, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.887945 18970 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 08:18:40.890363 18970 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:18:40.892146 14051 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 97d7768da26e4fff966e494446a278f2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER failed with leader 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 8 but the committed config has opid_index of 9
I20240502 08:18:40.894299 14598 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:40.905462 14062 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
W20240502 08:18:40.917205 14272 consensus_peers.cc:485] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e901787c37a40e9a7c6db44091d11ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:40.918850 19005 tablet_copy_client.cc:308] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:40.919783 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 2c1fecce883346459e21c29b98d95b7b from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:34765)
I20240502 08:18:40.919960 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 2c1fecce883346459e21c29b98d95b7b from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:34765: session id = 42116b63c31a436ca060c34691780d04-2c1fecce883346459e21c29b98d95b7b
I20240502 08:18:40.923617 14226 tablet_copy_source_session.cc:205] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:40.927719 16399 raft_consensus.cc:2932] T cf7124419bae40dfaaabdd2e673a3eb0 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:40.930474 18956 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:40.931447 19005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1fecce883346459e21c29b98d95b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:40.939200 14597 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:40.939683 19038 tablet_replica.cc:330] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: stopping tablet replica
I20240502 08:18:40.939889 19038 raft_consensus.cc:2229] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:40.940155 19038 raft_consensus.cc:2258] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:40.940467 14057 catalog_manager.cc:5484] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:40.948043 18813 ts_tablet_manager.cc:1888] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240502 08:18:40.948419 18813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76dcfec9be9f4f19a43d3f4603ae89d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:40.953013 18813 tablet_copy_client.cc:783] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:40.953261 18813 tablet_copy_client.cc:647] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:40.955255 14598 consensus_queue.cc:237] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:40.958748 18698 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:40.959069 16399 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:40.963575 19038 ts_tablet_manager.cc:1875] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:40.963909 16268 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.964154 16138 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:40.964756 18791 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.965976 16531 raft_consensus.cc:1261] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:40.966826 19041 consensus_queue.cc:1035] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:40.967803 18633 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.970209 16531 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.973295 16267 raft_consensus.cc:2932] T e6d3c011ad9546e3b69e23cbef587dbc P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.976769 14057 catalog_manager.cc:5484] T e6d3c011ad9546e3b69e23cbef587dbc P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 8 to 9, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:40.989244 14206 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.989812 16399 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:40.990448 19000 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:40.990871 18696 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:40.991299 14401 consensus_peers.cc:485] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b44cc3d7fc9944c7abf33b4da92b2f6f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:40.994925 18696 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:40.998430 19005 tablet_copy_client.cc:783] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:40.998730 19005 tablet_copy_client.cc:647] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:41.000192 14206 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:41.001345 14336 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.007282 16399 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:41.009553 19038 ts_tablet_manager.cc:1888] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 08:18:41.011679 14062 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 7 to 8, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:41.012399 14058 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 8 to 9, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.014914 14335 raft_consensus.cc:2932] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.021102 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6d3c011ad9546e3b69e23cbef587dbc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:41.028566 18813 tablet_copy_client.cc:523] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:41.031136 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d3c011ad9546e3b69e23cbef587dbc with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:41.035835 14315 tablet_service.cc:1514] Processing DeleteTablet for tablet b44cc3d7fc9944c7abf33b4da92b2f6f with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:41.042989 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b44cc3d7fc9944c7abf33b4da92b2f6f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:41.071452 19058 tablet_replica.cc:330] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:41.071938 19058 raft_consensus.cc:2229] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:41.072737 19058 raft_consensus.cc:2258] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:41.088554 14206 consensus_queue.cc:237] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } }
I20240502 08:18:41.100339 19007 tablet_copy_client.cc:308] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:41.101087 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 2c1fecce883346459e21c29b98d95b7b from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:40247)
I20240502 08:18:41.101284 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 2c1fecce883346459e21c29b98d95b7b from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:40247: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-2c1fecce883346459e21c29b98d95b7b
I20240502 08:18:41.104173 14226 tablet_copy_source_session.cc:205] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:41.106069 19007 ts_tablet_manager.cc:1875] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:41.107532 14469 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:41.013870 16533 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:41.052525 19005 tablet_copy_client.cc:523] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:41.110450 19038 log.cc:1196] T cf7124419bae40dfaaabdd2e673a3eb0 P 42116b63c31a436ca060c34691780d04: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/wals/cf7124419bae40dfaaabdd2e673a3eb0
I20240502 08:18:41.111249 16268 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:41.111502 14049 catalog_manager.cc:4830] TS 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): tablet cf7124419bae40dfaaabdd2e673a3eb0 (table rebalance_test_table_6 [id=9f670750f860417b897070d52d84d0c2]) successfully deleted
I20240502 08:18:41.113011 18706 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:41.114241 14401 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84eaabbe91524d129a170ad3519c265a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:41.114372 19005 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
W20240502 08:18:41.114825 14400 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84eaabbe91524d129a170ad3519c265a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:41.115063 14335 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:41.118125 18706 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:41.118718 19007 ts_tablet_manager.cc:1888] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 08:18:41.119060 19007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1fecce883346459e21c29b98d95b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:41.120337 19058 ts_tablet_manager.cc:1875] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:41.121402 18698 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:41.122488 19007 tablet_copy_client.cc:783] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:41.122790 19007 tablet_copy_client.cc:647] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:41.126716 14335 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:41.131099 16268 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:41.152254 14065 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:41.158219 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:41.162441 18813 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:41.168644 19058 ts_tablet_manager.cc:1888] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:41.172871 19058 log.cc:1196] T e6d3c011ad9546e3b69e23cbef587dbc P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/e6d3c011ad9546e3b69e23cbef587dbc
I20240502 08:18:41.176960 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet e6d3c011ad9546e3b69e23cbef587dbc (table rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]) successfully deleted
I20240502 08:18:41.182168 19007 tablet_copy_client.cc:523] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:41.175494 19058 tablet_replica.cc:330] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:41.182730 19058 raft_consensus.cc:2229] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:41.186695 19007 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:41.187274 19058 raft_consensus.cc:2258] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:41.188735 19058 ts_tablet_manager.cc:1875] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:41.196435 14141 consensus_peers.cc:485] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): Couldn't send request to peer 8ae1cf63e5bd46cf9a4ba89651971dde. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:41.199949 14469 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.200654 18959 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:41.213903 19058 ts_tablet_manager.cc:1888] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:18:41.215962 19058 log.cc:1196] T b44cc3d7fc9944c7abf33b4da92b2f6f P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/b44cc3d7fc9944c7abf33b4da92b2f6f
I20240502 08:18:41.218768 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet b44cc3d7fc9944c7abf33b4da92b2f6f (table rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]) successfully deleted
I20240502 08:18:41.224815 16268 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.226184 18959 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:41.254552 18959 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.260413 14466 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.261191 18806 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:41.261629 18806 tablet_bootstrap.cc:492] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:41.261925 18806 ts_tablet_manager.cc:1367] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.683s	user 0.058s	sys 0.001s
I20240502 08:18:41.262768 18806 raft_consensus.cc:348] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:41.263021 18806 raft_consensus.cc:729] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:41.263249 18806 consensus_queue.cc:260] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:41.264357 18806 ts_tablet_manager.cc:1398] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 08:18:41.265563 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-52331b69903943cb9d74f5ef5787dd00 received from {username='slave'} at 127.6.151.69:39627
I20240502 08:18:41.265712 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-52331b69903943cb9d74f5ef5787dd00 on tablet 52331b69903943cb9d74f5ef5787dd00 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:41.271854 14062 catalog_manager.cc:5484] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.288152 16138 raft_consensus.cc:1203] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:18:41.296088 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.301852 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:41.308213 18990 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:41.308771 18990 tablet_bootstrap.cc:492] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:41.309113 18990 ts_tablet_manager.cc:1367] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.565s	user 0.058s	sys 0.010s
I20240502 08:18:41.310338 18990 raft_consensus.cc:348] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:41.310770 18990 raft_consensus.cc:729] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:41.323653 18990 consensus_queue.cc:260] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:41.326679 18990 ts_tablet_manager.cc:1398] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.000s
I20240502 08:18:41.329406 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-6379601e2b2d4bdabde2773fcbe06fb4 received from {username='slave'} at 127.6.151.71:52741
I20240502 08:18:41.329564 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-6379601e2b2d4bdabde2773fcbe06fb4 on tablet 6379601e2b2d4bdabde2773fcbe06fb4 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:41.334892 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.355217 16268 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.362464 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.376034 18990 ts_tablet_manager.cc:902] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:41.379628 18806 ts_tablet_manager.cc:902] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:41.388823 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:41.395323 14335 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:41.396801 18806 tablet_copy_client.cc:308] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
W20240502 08:18:41.399744 14271 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:41.401027 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:41.403524 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 97d7768da26e4fff966e494446a278f2 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:39627)
I20240502 08:18:41.403919 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 97d7768da26e4fff966e494446a278f2 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:39627: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-97d7768da26e4fff966e494446a278f2
I20240502 08:18:41.413064 14357 tablet_copy_source_session.cc:205] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:41.414422 18806 ts_tablet_manager.cc:1875] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:41.417336 14598 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:41.422497 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:41.424324 18806 ts_tablet_manager.cc:1888] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:18:41.429402 18963 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.011s
I20240502 08:18:41.433401 14335 consensus_queue.cc:237] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:41.433497 18963 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.435880 14598 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.439385 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97d7768da26e4fff966e494446a278f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:41.445485 14062 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 9 to 10, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.450537 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 97d7768da26e4fff966e494446a278f2 with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:41.474452 18806 tablet_copy_client.cc:783] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:41.475816 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 97d7768da26e4fff966e494446a278f2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:41.476855 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:41.485981 14271 consensus_peers.cc:485] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96365c4348464db59463182aea8c1c23. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:41.487479 14598 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.488188 18956 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:41.488744 16400 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.489313 18956 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:41.495870 14597 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } }
I20240502 08:18:41.506558 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:41.511914 18806 tablet_copy_client.cc:647] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:41.516659 18806 tablet_copy_client.cc:523] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:41.521453 19132 ts_tablet_manager.cc:902] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:41.521539 18970 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.523931 14469 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.524704 18786 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:41.525035 14597 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.527437 16267 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.529263 16400 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.532907 14062 catalog_manager.cc:5484] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:41.542654 14141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 220027us
I20240502 08:18:41.528018 19041 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:18:41.546051 14535 consensus_peers.cc:485] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): Couldn't send request to peer 8ae1cf63e5bd46cf9a4ba89651971dde. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:41.550118 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:41.563797 18806 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:41.568806 16531 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:41.573226 18956 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:41.577523 16531 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:41.591431 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.600672 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.620265 19140 ts_tablet_manager.cc:902] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:41.620235 19041 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.623651 16267 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.637730 14057 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.640128 19143 ts_tablet_manager.cc:902] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:41.686630 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:41.710115 18990 tablet_copy_client.cc:308] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:41.710301 16400 raft_consensus.cc:1203] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:41.713002 19162 ts_tablet_manager.cc:902] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:41.714968 19128 tablet_replica.cc:330] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:41.715268 19128 raft_consensus.cc:2229] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:41.715973 19128 raft_consensus.cc:2258] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:41.727036 19128 ts_tablet_manager.cc:1875] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:41.732208 14597 consensus_queue.cc:237] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:41.747138 14469 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } }
I20240502 08:18:41.752583 18963 raft_consensus.cc:1050] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:41.753549 18963 consensus_queue.cc:237] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:41.757361 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 6e901787c37a40e9a7c6db44091d11ef from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:34397)
I20240502 08:18:41.757834 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 6e901787c37a40e9a7c6db44091d11ef from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:34397: session id = b3dfd9d6472342689afd86203f7afe5f-6e901787c37a40e9a7c6db44091d11ef
W20240502 08:18:41.760643 14535 consensus_peers.cc:485] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b -> Peer b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): Couldn't send request to peer b3dfd9d6472342689afd86203f7afe5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 810c5f8a81ba4e0ab06129c0178f12e7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:41.762328 16533 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.763228 14357 tablet_copy_source_session.cc:205] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:41.763237 18786 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:41.764681 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e901787c37a40e9a7c6db44091d11ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:41.775297 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:41.775727 18990 tablet_copy_client.cc:783] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:41.779706 18990 tablet_copy_client.cc:647] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:41.784794 16138 raft_consensus.cc:1261] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:41.785849 18956 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:41.790352 18990 tablet_copy_client.cc:523] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:41.791045 14597 raft_consensus.cc:1261] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:41.792805 16533 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:41.798871 14062 catalog_manager.cc:5484] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:41.799643 19132 tablet_copy_client.cc:308] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:41.801548 19143 tablet_copy_client.cc:308] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:41.792116 19041 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:41.802791 18956 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:18:41.811159 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 784f72f28102445f8c4c44a3e3da0d2d from peer 8ae1cf63e5bd46cf9a4ba89651971dde ({username='slave'} at 127.6.151.66:53413)
I20240502 08:18:41.811352 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 784f72f28102445f8c4c44a3e3da0d2d from peer 8ae1cf63e5bd46cf9a4ba89651971dde at {username='slave'} at 127.6.151.66:53413: session id = 8ae1cf63e5bd46cf9a4ba89651971dde-784f72f28102445f8c4c44a3e3da0d2d
I20240502 08:18:41.814442 14226 tablet_copy_source_session.cc:205] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:41.829164 19143 ts_tablet_manager.cc:1875] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:41.842384 14467 raft_consensus.cc:1261] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:18:41.843482 18970 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.846242 14466 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:41.847891 14598 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.854923 16138 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.861883 14059 catalog_manager.cc:5484] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 4 to 5, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:41.867640 19140 tablet_copy_client.cc:308] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:41.868454 14336 consensus_queue.cc:237] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:41.875373 16138 raft_consensus.cc:1261] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:41.876019 18963 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:41.878978 18956 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.880429 19143 ts_tablet_manager.cc:1888] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:18:41.880844 19143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 784f72f28102445f8c4c44a3e3da0d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:41.883754 16135 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.893028 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52331b69903943cb9d74f5ef5787dd00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:41.901340 19162 tablet_copy_client.cc:308] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:41.903613 19128 ts_tablet_manager.cc:1888] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:18:41.903340 14057 catalog_manager.cc:5484] T 52331b69903943cb9d74f5ef5787dd00 P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.904034 19128 log.cc:1196] T 97d7768da26e4fff966e494446a278f2 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/97d7768da26e4fff966e494446a278f2
I20240502 08:18:41.905647 14597 raft_consensus.cc:1261] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:18:41.909317 18963 consensus_queue.cc:1035] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:18:41.911504 14598 raft_consensus.cc:2932] T 52331b69903943cb9d74f5ef5787dd00 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:41.928529 18990 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:41.932873 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:41.937785 14270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 48540.4us
I20240502 08:18:41.952579 19143 tablet_copy_client.cc:783] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:41.952988 19143 tablet_copy_client.cc:647] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Starting download of 1 WAL segments...
W20240502 08:18:41.960803 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.71:39897, user_credentials={real_user=slave}} blocked reactor thread for 113724us
I20240502 08:18:41.960942 19143 tablet_copy_client.cc:523] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:41.971191 18633 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:41.972415 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet 97d7768da26e4fff966e494446a278f2 (table rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]) successfully deleted
I20240502 08:18:41.978724 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet b44cc3d7fc9944c7abf33b4da92b2f6f from peer 5b84b46faddb47fcae86cc5f3f478bf1 ({username='slave'} at 127.6.151.70:34071)
I20240502 08:18:41.978919 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet b44cc3d7fc9944c7abf33b4da92b2f6f from peer 5b84b46faddb47fcae86cc5f3f478bf1 at {username='slave'} at 127.6.151.70:34071: session id = 5b84b46faddb47fcae86cc5f3f478bf1-b44cc3d7fc9944c7abf33b4da92b2f6f
I20240502 08:18:41.982676 14489 tablet_copy_source_session.cc:205] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:41.994096 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 84eaabbe91524d129a170ad3519c265a from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:43863)
I20240502 08:18:41.994410 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 84eaabbe91524d129a170ad3519c265a from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:43863: session id = 42116b63c31a436ca060c34691780d04-84eaabbe91524d129a170ad3519c265a
I20240502 08:18:41.997874 14489 tablet_copy_source_session.cc:205] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:42.009159 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 52331b69903943cb9d74f5ef5787dd00 with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:42.010075 19140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84eaabbe91524d129a170ad3519c265a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:42.010082 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44cc3d7fc9944c7abf33b4da92b2f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:42.019012 19132 tablet_copy_client.cc:783] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:42.019385 19132 tablet_copy_client.cc:647] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:42.030162 14466 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } }
I20240502 08:18:42.039644 19132 tablet_copy_client.cc:523] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:42.040510 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:42.042763 19143 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:42.046984 19205 ts_tablet_manager.cc:902] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:42.048449 19132 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap starting.
I20240502 08:18:42.049916 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 84eaabbe91524d129a170ad3519c265a from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:44057)
I20240502 08:18:42.049232 19140 tablet_copy_client.cc:783] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:42.051952 19140 tablet_copy_client.cc:647] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:42.057406 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 84eaabbe91524d129a170ad3519c265a from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:44057: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-84eaabbe91524d129a170ad3519c265a
I20240502 08:18:42.065385 14469 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: true } }
I20240502 08:18:42.075222 19128 tablet_replica.cc:330] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:42.078454 14489 tablet_copy_source_session.cc:205] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:42.078724 19128 raft_consensus.cc:2229] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:42.081735 19162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84eaabbe91524d129a170ad3519c265a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:42.089404 19162 tablet_copy_client.cc:783] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:42.089790 19162 tablet_copy_client.cc:647] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:42.090281 19128 raft_consensus.cc:2258] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:42.091567 19162 tablet_copy_client.cc:523] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:42.092721 19128 ts_tablet_manager.cc:1875] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:42.095722 19162 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:42.110937 19191 ts_tablet_manager.cc:902] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
I20240502 08:18:42.112078 16400 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:42.113020 18583 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:42.129774 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:42.135574 14400 consensus_peers.cc:485] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Couldn't send request to peer b19bedd6f7f94bed9c727f6c7210cc8a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:42.157353 19187 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: true } } }
I20240502 08:18:42.165091 16400 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: true } } }
I20240502 08:18:42.165565 14065 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 9 to 10, NON_VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:42.177299 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:42.179839 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:42.200438 19128 ts_tablet_manager.cc:1888] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:18:42.200601 19128 log.cc:1196] T 52331b69903943cb9d74f5ef5787dd00 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/52331b69903943cb9d74f5ef5787dd00
I20240502 08:18:42.201104 16117 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:42.201401 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet 52331b69903943cb9d74f5ef5787dd00 (table rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]) successfully deleted
W20240502 08:18:42.245306 14141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.67:39913, user_credentials={real_user=slave}} blocked reactor thread for 139742us
I20240502 08:18:42.245630 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:42.270677 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:42.273871 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:42.300601 19205 tablet_copy_client.cc:308] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:42.306991 19140 tablet_copy_client.cc:523] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:42.312880 16533 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:42.313750 19195 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:42.318908 19140 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:42.353570 19111 raft_consensus.cc:1050] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:42.354948 19111 consensus_queue.cc:237] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:42.359658 19242 ts_tablet_manager.cc:902] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:42.367390 16138 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:42.370276 19111 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:42.375905 16400 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:42.376595 14597 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:42.376942 19111 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:42.377377 18964 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:42.384610 19214 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.385032 16400 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.391659 14065 catalog_manager.cc:5484] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.399914 14597 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.415227 19191 tablet_copy_client.cc:308] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:44269
I20240502 08:18:42.415980 19242 tablet_copy_client.cc:308] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:42.417228 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 810c5f8a81ba4e0ab06129c0178f12e7 from peer b3dfd9d6472342689afd86203f7afe5f ({username='slave'} at 127.6.151.71:40315)
I20240502 08:18:42.417418 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 810c5f8a81ba4e0ab06129c0178f12e7 from peer b3dfd9d6472342689afd86203f7afe5f at {username='slave'} at 127.6.151.71:40315: session id = b3dfd9d6472342689afd86203f7afe5f-810c5f8a81ba4e0ab06129c0178f12e7
I20240502 08:18:42.422458 14620 tablet_copy_source_session.cc:205] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:42.424121 19242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810c5f8a81ba4e0ab06129c0178f12e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:42.431800 19242 tablet_copy_client.cc:783] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:42.432132 19242 tablet_copy_client.cc:647] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:42.436497 19242 tablet_copy_client.cc:523] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:42.440117 19242 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap starting.
I20240502 08:18:42.457747 18806 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:18:42.466334 18806 tablet_bootstrap.cc:492] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:42.469894 16533 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: true } } }
I20240502 08:18:42.471004 18806 ts_tablet_manager.cc:1367] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 0.907s	user 0.065s	sys 0.000s
I20240502 08:18:42.474455 18806 raft_consensus.cc:348] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:42.480604 18813 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:42.481300 18806 raft_consensus.cc:729] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:42.487545 18806 consensus_queue.cc:260] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:42.493243 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 930e3af93722422fb9237fd9b623ceac from peer 8ae1cf63e5bd46cf9a4ba89651971dde ({username='slave'} at 127.6.151.66:51175)
I20240502 08:18:42.493449 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 930e3af93722422fb9237fd9b623ceac from peer 8ae1cf63e5bd46cf9a4ba89651971dde at {username='slave'} at 127.6.151.66:51175: session id = 8ae1cf63e5bd46cf9a4ba89651971dde-930e3af93722422fb9237fd9b623ceac
I20240502 08:18:42.493463 16138 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.496481 14620 tablet_copy_source_session.cc:205] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:42.497294 14206 consensus_queue.cc:237] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:42.500073 16400 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:42.500522 14357 tablet_copy_service.cc:140] P 8ae1cf63e5bd46cf9a4ba89651971dde: Received BeginTabletCopySession request for tablet 96365c4348464db59463182aea8c1c23 from peer f28bbf93da0c45eaaae0959e7a2f1e7e ({username='slave'} at 127.6.151.69:39627)
I20240502 08:18:42.500757 19110 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:42.510228 18964 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.517858 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6379601e2b2d4bdabde2773fcbe06fb4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:42.518352 14357 tablet_copy_service.cc:161] P 8ae1cf63e5bd46cf9a4ba89651971dde: Beginning new tablet copy session on tablet 96365c4348464db59463182aea8c1c23 from peer f28bbf93da0c45eaaae0959e7a2f1e7e at {username='slave'} at 127.6.151.69:39627: session id = f28bbf93da0c45eaaae0959e7a2f1e7e-96365c4348464db59463182aea8c1c23
I20240502 08:18:42.520237 14065 catalog_manager.cc:5484] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:42.521939 16400 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.524606 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 6379601e2b2d4bdabde2773fcbe06fb4 with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:42.532063 16134 raft_consensus.cc:1261] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:42.532881 19214 consensus_queue.cc:1035] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:42.537155 16134 raft_consensus.cc:2932] T 6379601e2b2d4bdabde2773fcbe06fb4 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:42.574215 19263 tablet_replica.cc:330] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:42.574555 19263 raft_consensus.cc:2229] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:42.575065 19263 raft_consensus.cc:2258] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:42.596050 19205 ts_tablet_manager.cc:1875] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:42.607945 19263 ts_tablet_manager.cc:1875] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:42.616906 14357 tablet_copy_source_session.cc:205] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:42.619256 19191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96365c4348464db59463182aea8c1c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:42.627028 19191 tablet_copy_client.cc:783] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:42.627091 19205 ts_tablet_manager.cc:1888] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:18:42.627362 19191 tablet_copy_client.cc:647] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:42.629007 19205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930e3af93722422fb9237fd9b623ceac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:42.630988 19191 tablet_copy_client.cc:523] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:42.634580 19191 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap starting.
I20240502 08:18:42.484392 19007 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:42.644483 18813 tablet_bootstrap.cc:492] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:42.644901 18813 ts_tablet_manager.cc:1367] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 1.483s	user 0.061s	sys 0.000s
I20240502 08:18:42.645766 18813 raft_consensus.cc:348] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:42.657915 18806 ts_tablet_manager.cc:1398] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.184s	user 0.005s	sys 0.000s
I20240502 08:18:42.668406 19007 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:42.668856 19007 ts_tablet_manager.cc:1367] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 1.482s	user 0.057s	sys 0.009s
I20240502 08:18:42.670445 19007 raft_consensus.cc:348] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:42.670994 19007 raft_consensus.cc:729] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:42.671317 19007 consensus_queue.cc:260] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:42.672008 19007 ts_tablet_manager.cc:1398] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:42.674508 19275 ts_tablet_manager.cc:902] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:42.674754 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-2c1fecce883346459e21c29b98d95b7b received from {username='slave'} at 127.6.151.69:40247
I20240502 08:18:42.674993 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-2c1fecce883346459e21c29b98d95b7b on tablet 2c1fecce883346459e21c29b98d95b7b with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:42.676921 18813 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:42.678277 18813 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:42.679491 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-97d7768da26e4fff966e494446a278f2 received from {username='slave'} at 127.6.151.69:39627
I20240502 08:18:42.679636 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-97d7768da26e4fff966e494446a278f2 on tablet 97d7768da26e4fff966e494446a278f2 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:42.684065 19205 tablet_copy_client.cc:783] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:42.695463 19205 tablet_copy_client.cc:647] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:42.699095 18813 ts_tablet_manager.cc:1398] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.054s	user 0.002s	sys 0.000s
I20240502 08:18:42.699900 19205 tablet_copy_client.cc:523] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:42.701292 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-76dcfec9be9f4f19a43d3f4603ae89d8 received from {username='slave'} at 127.6.151.70:38521
I20240502 08:18:42.701462 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-76dcfec9be9f4f19a43d3f4603ae89d8 on tablet 76dcfec9be9f4f19a43d3f4603ae89d8 with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:42.701493 19263 ts_tablet_manager.cc:1888] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:42.701637 19263 log.cc:1196] T 6379601e2b2d4bdabde2773fcbe06fb4 P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/6379601e2b2d4bdabde2773fcbe06fb4
I20240502 08:18:42.709484 14203 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:42.725118 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 6379601e2b2d4bdabde2773fcbe06fb4 (table rebalance_test_table_4 [id=e34a7e4087de424b9cab81bfbc828397]) successfully deleted
I20240502 08:18:42.674139 19132 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:42.739073 19132 tablet_bootstrap.cc:492] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: Bootstrap complete.
I20240502 08:18:42.739452 19132 ts_tablet_manager.cc:1367] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent bootstrapping tablet: real 0.691s	user 0.058s	sys 0.009s
I20240502 08:18:42.740334 19132 raft_consensus.cc:348] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:42.740623 19132 raft_consensus.cc:729] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Initialized, Role: LEARNER
I20240502 08:18:42.752847 19205 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap starting.
I20240502 08:18:42.758059 16268 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:42.758849 19214 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:42.740876 19132 consensus_queue.cc:260] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:42.766330 19275 tablet_copy_client.cc:308] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:42.768054 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 6272b2f3a9504e01832cb31f5b322980 from peer b19bedd6f7f94bed9c727f6c7210cc8a ({username='slave'} at 127.6.151.65:34957)
I20240502 08:18:42.768249 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 6272b2f3a9504e01832cb31f5b322980 from peer b19bedd6f7f94bed9c727f6c7210cc8a at {username='slave'} at 127.6.151.65:34957: session id = b19bedd6f7f94bed9c727f6c7210cc8a-6272b2f3a9504e01832cb31f5b322980
I20240502 08:18:42.765864 19132 ts_tablet_manager.cc:1398] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1: Time spent starting tablet: real 0.026s	user 0.001s	sys 0.001s
I20240502 08:18:42.774252 14489 tablet_copy_source_session.cc:205] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:42.774327 14488 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-b44cc3d7fc9944c7abf33b4da92b2f6f received from {username='slave'} at 127.6.151.70:34071
I20240502 08:18:42.774446 14488 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session 5b84b46faddb47fcae86cc5f3f478bf1-b44cc3d7fc9944c7abf33b4da92b2f6f on tablet b44cc3d7fc9944c7abf33b4da92b2f6f with peer 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:42.778972 19275 ts_tablet_manager.cc:1875] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:42.791566 16533 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:42.793057 18964 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:42.809291 14335 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:42.810420 19214 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240502 08:18:42.822767 18964 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.829422 19005 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:42.830032 19005 tablet_bootstrap.cc:492] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:42.830423 19005 ts_tablet_manager.cc:1367] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 1.716s	user 0.062s	sys 0.007s
I20240502 08:18:42.832219 14065 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 9 to 10. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:42.834818 16531 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.845575 14336 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.831547 19005 raft_consensus.cc:348] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:42.848897 19005 raft_consensus.cc:729] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:42.849220 19005 consensus_queue.cc:260] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:42.851094 19005 ts_tablet_manager.cc:1398] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.021s	user 0.003s	sys 0.000s
I20240502 08:18:42.852437 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-2c1fecce883346459e21c29b98d95b7b received from {username='slave'} at 127.6.151.72:34765
I20240502 08:18:42.852605 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 42116b63c31a436ca060c34691780d04-2c1fecce883346459e21c29b98d95b7b on tablet 2c1fecce883346459e21c29b98d95b7b with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:42.866394 14203 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:42.872236 16267 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:42.873883 19111 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:42.876689 16267 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.885593 16533 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:42.886406 19111 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:42.889547 18990 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:42.902874 19275 ts_tablet_manager.cc:1888] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 08:18:42.903311 19275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6272b2f3a9504e01832cb31f5b322980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:42.908387 16138 raft_consensus.cc:1203] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 08:18:42.925598 18964 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.929195 16531 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.935684 14057 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 10 to 11, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.936115 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f0bcdf713f124d46a22ddfc2873cd742 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:42.940754 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet f0bcdf713f124d46a22ddfc2873cd742 with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:42.946234 14203 consensus_queue.cc:237] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:42.947240 16268 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:42.949743 18990 tablet_bootstrap.cc:492] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:42.950147 18990 ts_tablet_manager.cc:1367] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 1.022s	user 0.056s	sys 0.004s
I20240502 08:18:42.951269 18990 raft_consensus.cc:348] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:42.951561 18990 raft_consensus.cc:729] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:42.951834 18990 consensus_queue.cc:260] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:42.952651 18990 ts_tablet_manager.cc:1398] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:18:42.956451 19275 tablet_copy_client.cc:783] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:42.957396 19275 tablet_copy_client.cc:647] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:42.961383 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-6e901787c37a40e9a7c6db44091d11ef received from {username='slave'} at 127.6.151.71:34397
I20240502 08:18:42.961563 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-6e901787c37a40e9a7c6db44091d11ef on tablet 6e901787c37a40e9a7c6db44091d11ef with peer b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:42.971071 14140 consensus_peers.cc:485] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0bcdf713f124d46a22ddfc2873cd742. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:42.973717 16531 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:42.974283 19111 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:42.975261 19275 tablet_copy_client.cc:523] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:42.977280 19111 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:42.979900 19275 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap starting.
I20240502 08:18:42.981734 19304 tablet_replica.cc:330] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:42.982414 14049 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f0bcdf713f124d46a22ddfc2873cd742 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:42.982775 16531 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:42.985531 19304 raft_consensus.cc:2229] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:42.985910 19304 raft_consensus.cc:2258] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:42.985037 14065 catalog_manager.cc:5484] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 11 to 12, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:42.993222 19140 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:42.993582 16138 raft_consensus.cc:1203] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 08:18:42.993810 19140 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:42.994174 19140 ts_tablet_manager.cc:1367] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.675s	user 0.069s	sys 0.003s
I20240502 08:18:42.993902 16138 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:42.994946 19304 ts_tablet_manager.cc:1875] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:43.017271 19140 raft_consensus.cc:348] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:43.017733 19140 raft_consensus.cc:729] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:43.018018 19140 consensus_queue.cc:260] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:43.024611 19140 ts_tablet_manager.cc:1398] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.000s
I20240502 08:18:43.025591 16267 raft_consensus.cc:1261] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:43.026348 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-84eaabbe91524d129a170ad3519c265a received from {username='slave'} at 127.6.151.72:43863
I20240502 08:18:43.026477 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session 42116b63c31a436ca060c34691780d04-84eaabbe91524d129a170ad3519c265a on tablet 84eaabbe91524d129a170ad3519c265a with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:43.026564 18964 consensus_queue.cc:1035] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:43.028791 16267 raft_consensus.cc:2932] T f0bcdf713f124d46a22ddfc2873cd742 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:43.041007 14469 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } }
I20240502 08:18:43.066555 16400 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:43.066689 16134 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:43.067883 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:43.068662 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:43.074733 19304 ts_tablet_manager.cc:1888] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 08:18:43.075367 19304 log.cc:1196] T f0bcdf713f124d46a22ddfc2873cd742 P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0bcdf713f124d46a22ddfc2873cd742
I20240502 08:18:43.077872 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet f0bcdf713f124d46a22ddfc2873cd742 (table rebalance_test_table_0 [id=f5e582fa17054f269c52c3a87841b87c]) successfully deleted
I20240502 08:18:43.089026 14206 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:43.089848 18583 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:43.108009 16267 raft_consensus.cc:1203] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:18:43.117738 19242 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:43.118407 19242 tablet_bootstrap.cc:492] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: Bootstrap complete.
I20240502 08:18:43.118825 19242 ts_tablet_manager.cc:1367] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: Time spent bootstrapping tablet: real 0.679s	user 0.103s	sys 0.025s
I20240502 08:18:43.119865 19242 raft_consensus.cc:348] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:43.120505 19242 raft_consensus.cc:729] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Initialized, Role: LEARNER
I20240502 08:18:43.120802 19242 consensus_queue.cc:260] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }
I20240502 08:18:43.123154 19242 ts_tablet_manager.cc:1398] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 08:18:43.124028 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session b3dfd9d6472342689afd86203f7afe5f-810c5f8a81ba4e0ab06129c0178f12e7 received from {username='slave'} at 127.6.151.71:40315
I20240502 08:18:43.124259 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session b3dfd9d6472342689afd86203f7afe5f-810c5f8a81ba4e0ab06129c0178f12e7 on tablet 810c5f8a81ba4e0ab06129c0178f12e7 with peer b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:43.132651 19195 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } }
I20240502 08:18:43.137622 16532 raft_consensus.cc:1203] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:18:43.146414 16400 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } }
I20240502 08:18:43.155884 14057 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 9 to 10. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } }
I20240502 08:18:43.162173 16138 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } }
I20240502 08:18:43.165884 14206 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } }
I20240502 08:18:43.173472 14469 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
W20240502 08:18:43.177923 14400 consensus_peers.cc:485] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f109afd932d4c77a6b7cd63c417bc70. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:43.184638 16399 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.185525 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.189627 14203 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.190335 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.193981 19187 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.201277 14051 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4f109afd932d4c77a6b7cd63c417bc70 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:18:43.204120 16400 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.213989 14203 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.215531 19275 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:43.216068 19275 tablet_bootstrap.cc:492] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Bootstrap complete.
I20240502 08:18:43.216478 19275 ts_tablet_manager.cc:1367] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent bootstrapping tablet: real 0.237s	user 0.050s	sys 0.015s
I20240502 08:18:43.217195 14057 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 10 to 11, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:43.217479 19275 raft_consensus.cc:348] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: true } }
I20240502 08:18:43.217774 19275 raft_consensus.cc:729] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Initialized, Role: LEARNER
I20240502 08:18:43.218006 19275 consensus_queue.cc:260] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: true } }
I20240502 08:18:43.222177 19275 ts_tablet_manager.cc:1398] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 08:18:43.226230 16268 raft_consensus.cc:1203] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11]   Dedup: 1.11->[]
I20240502 08:18:43.226418 16134 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.227277 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.230306 16135 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.242491 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session b19bedd6f7f94bed9c727f6c7210cc8a-6272b2f3a9504e01832cb31f5b322980 received from {username='slave'} at 127.6.151.65:34957
I20240502 08:18:43.242700 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session b19bedd6f7f94bed9c727f6c7210cc8a-6272b2f3a9504e01832cb31f5b322980 on tablet 6272b2f3a9504e01832cb31f5b322980 with peer b19bedd6f7f94bed9c727f6c7210cc8a
I20240502 08:18:43.270447 14203 consensus_queue.cc:237] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:43.273325 16400 raft_consensus.cc:1203] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:43.275926 16531 raft_consensus.cc:1203] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 08:18:43.281504 14469 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:43.282291 19111 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:18:43.284853 14140 consensus_peers.cc:485] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b15694b0b284ecebd899522dc69d422. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:43.285393 16268 raft_consensus.cc:1261] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:43.285719 18964 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:43.289597 14469 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:43.292935 19214 consensus_queue.cc:1035] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:43.293504 14059 catalog_manager.cc:5484] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:43.309504 16400 raft_consensus.cc:1203] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:43.323601 18964 raft_consensus.cc:1050] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:43.325138 18964 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:43.379635 16268 raft_consensus.cc:2932] T 0b15694b0b284ecebd899522dc69d422 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:43.380868 18970 raft_consensus.cc:1050] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:43.381765 18970 consensus_queue.cc:237] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:43.403623 14597 consensus_queue.cc:237] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:43.408043 14335 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.408043 16531 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.409125 16268 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.409148 16135 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.382596 19162 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:43.415002 19162 tablet_bootstrap.cc:492] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:43.416254 19111 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.418246 19214 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.418865 19214 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.419267 19214 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.422381 19162 ts_tablet_manager.cc:1367] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 1.320s	user 0.060s	sys 0.005s
I20240502 08:18:43.431982 19111 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.444880 16268 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.453155 16531 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.425889 19162 raft_consensus.cc:348] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:43.458766 19162 raft_consensus.cc:729] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:43.459010 19162 consensus_queue.cc:260] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:43.458909 14059 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.457456 16135 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.460578 19162 ts_tablet_manager.cc:1398] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.038s	user 0.002s	sys 0.000s
I20240502 08:18:43.461433 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-84eaabbe91524d129a170ad3519c265a received from {username='slave'} at 127.6.151.69:44057
I20240502 08:18:43.461591 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-84eaabbe91524d129a170ad3519c265a on tablet 84eaabbe91524d129a170ad3519c265a with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:43.465886 19214 raft_consensus.cc:1050] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:43.466753 19214 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:43.472405 16267 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:43.475136 19110 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:43.475680 16533 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:43.478688 19110 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:43.480757 16269 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:43.480806 16134 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.481406 19041 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:43.482017 18963 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 08:18:43.483389 14533 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88890b0913c0462bade8208dfa0f64b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:43.484220 14334 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:43.484948 18633 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:43.486063 16533 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.486649 18963 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.491586 14597 raft_consensus.cc:1261] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:43.492570 18963 consensus_queue.cc:1035] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.492719 19143 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:43.493221 19143 tablet_bootstrap.cc:492] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap complete.
I20240502 08:18:43.493521 19143 ts_tablet_manager.cc:1367] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 1.451s	user 0.061s	sys 0.003s
I20240502 08:18:43.494354 19143 raft_consensus.cc:348] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } }
I20240502 08:18:43.504464 19162 ts_tablet_manager.cc:902] T f0bcdf713f124d46a22ddfc2873cd742 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
F20240502 08:18:43.504823 19162 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.queue_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.queue_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_cpu_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_cpu_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_wall_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_wall_time_us) (tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.queue_time_us, tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.queue_time_us) (tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_cpu_time_us, tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_cpu_time_us) (tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_wall_time_us, tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_wall_time_us) (tablet-download-pool-52331b69903943cb9d74f5ef5787dd00.queue_time_us, tablet-download-pool-52331b69903943cb9d74f5ef5787dd00.queue_time_us) (tablet-download-pool-52331b69903943cb9d74f5ef5787dd00.run_cpu_time_us, tablet-download-pool-52331b69903943cb9d74f5ef5787dd00.run_cpu_time_us) (tablet-download-pool-52331b69903943cb9d74f5ef5787dd00.run_wall_time_us, tablet-download-pool-52331b69903943cb9d74f5ef5787dd00.run_wall_time_us) (tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.queue_time_us, tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.queue_time_us) (tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_cpu_time_us, tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_cpu_time_us) (tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_wall_time_us, tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_wall_time_us) (tablet-download-pool-6379601e2b2d4bdabde2773fcbe06fb4.queue_time_us, tablet-download-pool-6379601e2b2d4bdabde2773fcbe06fb4.queue_time_us) (tablet-download-pool-6379601e2b2d4bdabde2773fcbe06fb4.run_cpu_time_us, tablet-download-pool-6379601e2b2d4bdabde2773fcbe06fb4.run_cpu_time_us) (tablet-download-pool-6379601e2b2d4bdabde2773fcbe06fb4.run_wall_time_us, tablet-download-pool-6379601e2b2d4bdabde2773fcbe06fb4.run_wall_time_us) (tablet-download-pool-7a92e7e903594d84a59f3ed058a5a3ec.queue_time_us, tablet-download-pool-7a92e7e903594d84a59f3ed058a5a3ec.queue_time_us) (tablet-download-pool-7a92e7e903594d84a59f3ed058a5a3ec.run_cpu_time_us, tablet-download-pool-7a92e7e903594d84a59f3ed058a5a3ec.run_cpu_time_us) (tablet-download-pool-7a92e7e903594d84a59f3ed058a5a3ec.run_wall_time_us, tablet-download-pool-7a92e7e903594d84a59f3ed058a5a3ec.run_wall_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.queue_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.queue_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_cpu_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_cpu_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_wall_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_wall_time_us) (tablet-download-pool-96365c4348464db59463182aea8c1c23.queue_time_us, tablet-download-pool-96365c4348464db59463182aea8c1c23.queue_time_us) (tablet-download-pool-96365c4348464db59463182aea8c1c23.run_cpu_time_us, tablet-download-pool-96365c4348464db59463182aea8c1c23.run_cpu_time_us) (tablet-download-pool-96365c4348464db59463182aea8c1c23.run_wall_time_us, tablet-download-pool-96365c4348464db59463182aea8c1c23.run_wall_time_us) (tablet-download-pool-97d7768da26e4fff966e494446a278f2.queue_time_us, tablet-download-pool-97d7768da26e4fff966e494446a278f2.queue_time_us) (tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_cpu_time_us, tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_cpu_time_us) (tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_wall_time_us, tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_wall_time_us) (tablet-download-pool-9ee2cdcfd53a469c803f846458dd083b.queue_time_us, tablet-download-pool-9ee2cdcfd53a469c803f846458dd083b.queue_time_us) (tablet-download-pool-9ee2cdcfd53a469c803f846458dd083b.run_cpu_time_us, tablet-download-pool-9ee2cdcfd53a469c803f846458dd083b.run_cpu_time_us) (tablet-download-pool-9ee2cdcfd53a469c803f846458dd083b.run_wall_time_us, tablet-download-pool-9ee2cdcfd53a469c803f846458dd083b.run_wall_time_us) (tablet-download-pool-a635dc6d64724295be4b4a7f68d0fad6.queue_time_us, tablet-download-pool-a635dc6d64724295be4b4a7f68d0fad6.queue_time_us) (tablet-download-pool-a635dc6d64724295be4b4a7f68d0fad6.run_cpu_time_us, tablet-download-pool-a635dc6d64724295be4b4a7f68d0fad6.run_cpu_time_us) (tablet-download-pool-a635dc6d64724295be4b4a7f68d0fad6.run_wall_time_us, tablet-download-pool-a635dc6d64724295be4b4a7f68d0fad6.run_wall_time_us) (tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.queue_time_us, tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.queue_time_us) (tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_cpu_time_us, tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_cpu_time_us) (tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_wall_time_us, tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_wall_time_us) (tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.queue_time_us, tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.queue_time_us) (tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_cpu_time_us, tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_cpu_time_us) (tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_wall_time_us, tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_wall_time_us) (tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.queue_time_us, tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.queue_time_us) (tablet-download-pool-fec3033140544f4994057d5b22973fff.queue_time_us, tablet-download-pool-fec3033140544f4994057d5b22973fff.queue_time_us) (tablet-download-pool-fec3033140544f4994057d5b22973fff.run_cpu_time_us, tablet-download-pool-fec3033140544f4994057d5b22973fff.run_cpu_time_us) (tablet-download-pool-fec3033140544f4994057d5b22973fff.run_wall_time_us, tablet-download-pool-fec3033140544f4994057d5b22973fff.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240502 08:18:43.514861 16138 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:43.516096 18964 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:43.517653 19143 raft_consensus.cc:729] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: LEARNER
I20240502 08:18:43.517972 19143 consensus_queue.cc:260] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } }
I20240502 08:18:43.518824 19143 ts_tablet_manager.cc:1398] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.025s	user 0.002s	sys 0.003s
I20240502 08:18:43.518970 19205 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:43.519493 19205 tablet_bootstrap.cc:492] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Bootstrap complete.
I20240502 08:18:43.519882 19205 ts_tablet_manager.cc:1367] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent bootstrapping tablet: real 0.767s	user 0.069s	sys 0.003s
I20240502 08:18:43.520946 19205 raft_consensus.cc:348] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } }
I20240502 08:18:43.521296 19205 raft_consensus.cc:729] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Initialized, Role: LEARNER
I20240502 08:18:43.521544 19205 consensus_queue.cc:260] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } }
I20240502 08:18:43.522531 19205 ts_tablet_manager.cc:1398] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde: Time spent starting tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:18:43.521981 19111 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.523777 18963 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:43.530002 16138 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.531872 16135 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:43.535916 14058 catalog_manager.cc:5484] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:43.536990 14062 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 11 to 12, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
    @     0x7f3e9b394dcd  google::LogMessage::Fail() at ??:0
    @     0x7f3e9b398b93  google::LogMessage::SendToLog() at ??:0
I20240502 08:18:43.551144 19041 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.551637 14598 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:43.558468 14337 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.561591 16267 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.579620 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session 8ae1cf63e5bd46cf9a4ba89651971dde-784f72f28102445f8c4c44a3e3da0d2d received from {username='slave'} at 127.6.151.66:53413
I20240502 08:18:43.579833 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session 8ae1cf63e5bd46cf9a4ba89651971dde-784f72f28102445f8c4c44a3e3da0d2d on tablet 784f72f28102445f8c4c44a3e3da0d2d with peer 8ae1cf63e5bd46cf9a4ba89651971dde
I20240502 08:18:43.581148 14336 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:43.582000 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 8ae1cf63e5bd46cf9a4ba89651971dde-930e3af93722422fb9237fd9b623ceac received from {username='slave'} at 127.6.151.66:51175
I20240502 08:18:43.582144 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 8ae1cf63e5bd46cf9a4ba89651971dde-930e3af93722422fb9237fd9b623ceac on tablet 930e3af93722422fb9237fd9b623ceac with peer 8ae1cf63e5bd46cf9a4ba89651971dde
I20240502 08:18:43.582978 18964 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:43.584455 16533 raft_consensus.cc:2932] T 97d7768da26e4fff966e494446a278f2 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:43.585551 14336 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.597661 14336 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.600762 18583 raft_consensus.cc:1050] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
I20240502 08:18:43.601634 18583 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:43.608886 19383 ts_tablet_manager.cc:902] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913)
I20240502 08:18:43.619696 14064 catalog_manager.cc:5484] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.622359 14597 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:43.629673 16267 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.630362 19195 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:43.631542 14203 consensus_queue.cc:237] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:43.635509 16135 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 08:18:43.638257 16400 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.639034 18583 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:43.640334 19111 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:18:43.645593 14203 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.646364 18583 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:43.555843 16531 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.665424 19383 tablet_copy_client.cc:308] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:39913
I20240502 08:18:43.665874 19243 raft_consensus.cc:1050] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:43.666935 19243 consensus_queue.cc:237] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:43.668521 14489 tablet_copy_service.cc:140] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Received BeginTabletCopySession request for tablet 4f109afd932d4c77a6b7cd63c417bc70 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:43863)
I20240502 08:18:43.668704 14489 tablet_copy_service.cc:161] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Beginning new tablet copy session on tablet 4f109afd932d4c77a6b7cd63c417bc70 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:43863: session id = 42116b63c31a436ca060c34691780d04-4f109afd932d4c77a6b7cd63c417bc70
I20240502 08:18:43.672555 14489 tablet_copy_source_session.cc:205] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:43.672775 14206 raft_consensus.cc:1203] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 08:18:43.674221 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f109afd932d4c77a6b7cd63c417bc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:43.687420 19383 tablet_copy_client.cc:783] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:43.687727 19383 tablet_copy_client.cc:647] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:43.687970 19395 ts_tablet_manager.cc:902] T 0b15694b0b284ecebd899522dc69d422 P f28bbf93da0c45eaaae0959e7a2f1e7e: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:43.688592 16399 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:43.689291 18633 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.692914 19383 tablet_copy_client.cc:523] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:43.696452 19383 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:43.697099 16532 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:43.697715 18633 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.697871 19187 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:43.709003 14064 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:43.709615 14469 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:43.709944 14203 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:43.710718 19243 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.712965 16401 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:43.734701 16135 raft_consensus.cc:1203] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:18:43.747431 19407 raft_consensus.cc:1050] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:43.748369 19407 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:43.765723 14337 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:18:43.781611 14533 consensus_peers.cc:485] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930e3af93722422fb9237fd9b623ceac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:43.781971 19243 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.811220 16530 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.811970 16269 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.812083 16135 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.813419 18583 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:43.814013 18583 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:43.814437 18583 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:43.817962 14467 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
W20240502 08:18:43.820616 14400 consensus_peers.cc:485] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:43.820657 14203 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:43.821226 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:43.821424 16138 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:43.822077 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:43.823446 16533 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 08:18:43.824708 19363 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:18:43.825075 16138 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.827313 14335 raft_consensus.cc:1261] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 08:18:43.828164 19110 consensus_queue.cc:1035] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:18:43.831768 19110 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.832527 19041 raft_consensus.cc:1050] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 to VOTER
W20240502 08:18:43.833225 14535 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.66:44269, user_credentials={real_user=slave}} blocked reactor thread for 66865.3us
I20240502 08:18:43.834017 14467 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:43.835606 16529 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.836122 19243 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.836653 19195 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.839270 14203 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.841562 18633 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:43.846549 18583 raft_consensus.cc:1013] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: attempt to promote peer f28bbf93da0c45eaaae0959e7a2f1e7e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 08:18:43.848722 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c1fecce883346459e21c29b98d95b7b with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:43.847903 14065 catalog_manager.cc:5484] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:43.850625 16135 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.852479 14062 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 11 to 12, VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.853103 14062 catalog_manager.cc:5484] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 12 to 13, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:18:43.849969 14534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.70:45703, user_credentials={real_user=slave}} blocked reactor thread for 126675us
I20240502 08:18:43.855329 16267 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:43.861596 14334 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.867759 19187 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:43.871040 19191 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:43.871843 19191 tablet_bootstrap.cc:492] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: Bootstrap complete.
I20240502 08:18:43.870306 14334 raft_consensus.cc:2932] T 2c1fecce883346459e21c29b98d95b7b P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:43.872422 19191 ts_tablet_manager.cc:1367] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent bootstrapping tablet: real 1.238s	user 0.064s	sys 0.004s
I20240502 08:18:43.873576 19191 raft_consensus.cc:348] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:43.873972 19191 raft_consensus.cc:729] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f28bbf93da0c45eaaae0959e7a2f1e7e, State: Initialized, Role: LEARNER
I20240502 08:18:43.874316 19191 consensus_queue.cc:260] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:43.883430 16382 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f109afd932d4c77a6b7cd63c417bc70 with delete_type TABLET_DATA_TOMBSTONED (TS b3dfd9d6472342689afd86203f7afe5f not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:53026
I20240502 08:18:43.884593 19262 raft_consensus.cc:1050] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER b3dfd9d6472342689afd86203f7afe5f to VOTER
I20240502 08:18:43.885597 19262 consensus_queue.cc:237] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:43.888653 16532 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
    @     0x7f3e9b3947cc  google::LogMessage::Flush() at ??:0
    @     0x7f3e9b395f59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:18:43.899843 16399 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:43.900485 18963 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.904803 14598 consensus_queue.cc:237] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:43.908316 19422 tablet_replica.cc:330] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: stopping tablet replica
I20240502 08:18:43.908519 19422 raft_consensus.cc:2229] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:43.908829 19422 raft_consensus.cc:2258] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:43.909938 19422 ts_tablet_manager.cc:1875] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:43.924500 18633 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.932792 14336 raft_consensus.cc:1203] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:43.939028 19041 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:43.942696 16402 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:43.945139 16399 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:43.958549 16268 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:43.966501 14598 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:43.968112 18963 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:43.968210 18633 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:43.973434 19423 log.cc:924] Time spent T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20240502 08:18:43.987262 19422 ts_tablet_manager.cc:1888] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 08:18:43.987437 19422 log.cc:1196] T 4f109afd932d4c77a6b7cd63c417bc70 P b3dfd9d6472342689afd86203f7afe5f: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-6/wal/wals/4f109afd932d4c77a6b7cd63c417bc70
I20240502 08:18:44.005045 14051 catalog_manager.cc:4830] TS b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897): tablet 4f109afd932d4c77a6b7cd63c417bc70 (table rebalance_test_table_3 [id=291881f82dc743a9954be49c6437402f]) successfully deleted
I20240502 08:18:44.023995 19405 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.034976 18583 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.046737 19383 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:44.048189 16267 raft_consensus.cc:2932] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:44.034068 16268 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.053548 16269 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.055490 19383 tablet_bootstrap.cc:492] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:44.063907 19383 ts_tablet_manager.cc:1367] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.359s	user 0.055s	sys 0.004s
I20240502 08:18:44.065013 19383 raft_consensus.cc:348] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:44.083662 19383 raft_consensus.cc:729] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:44.083993 19383 consensus_queue.cc:260] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:44.091563 16531 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.091778 19191 ts_tablet_manager.cc:1398] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e: Time spent starting tablet: real 0.039s	user 0.002s	sys 0.000s
I20240502 08:18:44.092783 14357 tablet_copy_service.cc:342] P 8ae1cf63e5bd46cf9a4ba89651971dde: Request end of tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-96365c4348464db59463182aea8c1c23 received from {username='slave'} at 127.6.151.69:39627
I20240502 08:18:44.092927 14357 tablet_copy_service.cc:433] P 8ae1cf63e5bd46cf9a4ba89651971dde: ending tablet copy session f28bbf93da0c45eaaae0959e7a2f1e7e-96365c4348464db59463182aea8c1c23 on tablet 96365c4348464db59463182aea8c1c23 with peer f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:44.103605 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f109afd932d4c77a6b7cd63c417bc70 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:44.112509 16137 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.117458 14059 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:44.123098 19405 raft_consensus.cc:1050] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b: attempting to promote NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde to VOTER
I20240502 08:18:44.123733 14203 consensus_queue.cc:237] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } }
I20240502 08:18:44.124219 19405 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:44.134205 16267 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
    @     0x7f3e9e0d5742  kudu::TraceMetrics::InternName() at ??:0
I20240502 08:18:44.145155 19243 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20240502 08:18:44.150628 14141 consensus_peers.cc:485] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): Couldn't send request to peer e79a5862de19490b89a9ea97860d776b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:44.150700 16133 raft_consensus.cc:1203] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:18:44.152001 16133 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:44.152750 14467 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:44.153460 19214 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:44.154011 19214 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:44.154325 19406 raft_consensus.cc:1050] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:44.155388 19406 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:44.159003 16133 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:44.159601 19187 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:44.161291 14334 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:44.161888 19406 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:44.163121 14334 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.163687 14059 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 8 to 9, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.169067 19187 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.173081 19262 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:44.173158 19110 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:44.180136 16267 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:44.182849 16133 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
W20240502 08:18:44.186339 14533 consensus_peers.cc:485] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930e3af93722422fb9237fd9b623ceac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:44.185905 14334 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.188691 14467 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:44.192008 19383 ts_tablet_manager.cc:1398] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.021s	user 0.002s	sys 0.000s
I20240502 08:18:44.196614 19404 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:44.201052 14489 tablet_copy_service.cc:342] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-4f109afd932d4c77a6b7cd63c417bc70 received from {username='slave'} at 127.6.151.72:43863
I20240502 08:18:44.201223 14489 tablet_copy_service.cc:433] P 32e074cd0dfd4311adc4e2a0f5d1ad8a: ending tablet copy session 42116b63c31a436ca060c34691780d04-4f109afd932d4c77a6b7cd63c417bc70 on tablet 4f109afd932d4c77a6b7cd63c417bc70 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:44.202672 14063 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 9 to 10, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.209704 16527 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:44.217319 16137 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:44.219882 18633 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:44.221766 14467 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.221735 14334 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.227907 16249 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1fecce883346459e21c29b98d95b7b with delete_type TABLET_DATA_TOMBSTONED (TS 5b84b46faddb47fcae86cc5f3f478bf1 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:48900
I20240502 08:18:44.231267 14063 catalog_manager.cc:5484] T 7a92e7e903594d84a59f3ed058a5a3ec P f28bbf93da0c45eaaae0959e7a2f1e7e reported cstate change: config changed from index 6 to 7, NON_VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } } }
I20240502 08:18:44.233361 19383 ts_tablet_manager.cc:902] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:44.236539 18970 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.237659 16402 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:44.238348 19409 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.239527 14467 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:44.243130 16402 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
    @     0x7f3e9e0a1f8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:18:44.252606 16267 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:44.253778 19406 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:44.255795 16267 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.261435 16400 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.261643 14335 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:44.263074 14064 catalog_manager.cc:5484] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.266049 16533 raft_consensus.cc:1261] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:44.267554 19405 consensus_queue.cc:1035] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:44.269436 14598 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.273907 18633 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:44.274580 18633 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:44.275070 18633 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:44.277848 19407 raft_consensus.cc:1050] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER b19bedd6f7f94bed9c727f6c7210cc8a to VOTER
I20240502 08:18:44.278175 14064 catalog_manager.cc:5484] T 810c5f8a81ba4e0ab06129c0178f12e7 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.278864 19407 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } }
I20240502 08:18:44.282799 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 810c5f8a81ba4e0ab06129c0178f12e7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:44.291805 14203 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:44.294322 18583 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:44.298044 16526 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:44.298120 16402 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:44.297602 19404 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.298892 18583 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:44.300624 18583 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:44.301736 16530 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.304351 16531 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:44.305502 19187 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:44.313279 14467 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.317128 19195 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:44.308836 16532 raft_consensus.cc:2932] T 810c5f8a81ba4e0ab06129c0178f12e7 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.323766 14336 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
    @     0x7f3e9e09bcd6  kudu::ThreadPoolBuilder::Build() at ??:0
I20240502 08:18:44.351490 19410 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:44.337422 16399 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:44.324616 19363 raft_consensus.cc:1050] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER 8ae1cf63e5bd46cf9a4ba89651971dde to VOTER
I20240502 08:18:44.386374 19363 consensus_queue.cc:237] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } }
I20240502 08:18:44.335525 14203 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:44.332237 19464 tablet_replica.cc:330] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: stopping tablet replica
I20240502 08:18:44.390049 19464 raft_consensus.cc:2229] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:44.390492 19464 raft_consensus.cc:2258] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:44.372654 16400 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:44.394361 19464 ts_tablet_manager.cc:1875] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:44.336098 16268 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.407152 14336 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.407881 19110 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:44.410912 16269 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.411628 19110 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:44.432493 14449 tablet_service.cc:1514] Processing DeleteTablet for tablet 810c5f8a81ba4e0ab06129c0178f12e7 with delete_type TABLET_DATA_TOMBSTONED (TS 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59708
I20240502 08:18:44.326493 16531 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:44.330499 16529 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.357890 14059 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 8 to 9, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.403126 14064 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 10 to 11, b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:44.353155 16528 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:44.359630 16530 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:44.425092 14598 consensus_queue.cc:237] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:44.368283 19383 tablet_copy_client.cc:308] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.68:42587
I20240502 08:18:44.419106 16527 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b3dfd9d6472342689afd86203f7afe5f (127.6.151.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.456593 14620 tablet_copy_service.cc:140] P e79a5862de19490b89a9ea97860d776b: Received BeginTabletCopySession request for tablet 88890b0913c0462bade8208dfa0f64b7 from peer 42116b63c31a436ca060c34691780d04 ({username='slave'} at 127.6.151.72:49117)
I20240502 08:18:44.456785 14620 tablet_copy_service.cc:161] P e79a5862de19490b89a9ea97860d776b: Beginning new tablet copy session on tablet 88890b0913c0462bade8208dfa0f64b7 from peer 42116b63c31a436ca060c34691780d04 at {username='slave'} at 127.6.151.72:49117: session id = 42116b63c31a436ca060c34691780d04-88890b0913c0462bade8208dfa0f64b7
I20240502 08:18:44.456902 14467 raft_consensus.cc:1261] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.458446 19111 consensus_queue.cc:1035] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:44.423542 19464 ts_tablet_manager.cc:1888] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 08:18:44.459740 19464 log.cc:1196] T 2c1fecce883346459e21c29b98d95b7b P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/wals/2c1fecce883346459e21c29b98d95b7b
I20240502 08:18:44.460839 14050 catalog_manager.cc:4830] TS 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): tablet 2c1fecce883346459e21c29b98d95b7b (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:44.462028 14620 tablet_copy_source_session.cc:205] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:44.463762 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88890b0913c0462bade8208dfa0f64b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:44.465358 14533 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88890b0913c0462bade8208dfa0f64b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:44.465976 14336 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.466614 19409 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:44.468605 14534 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88890b0913c0462bade8208dfa0f64b7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:44.469533 19409 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:44.471709 14335 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:44.427367 16267 raft_consensus.cc:2932] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:44.474761 16268 raft_consensus.cc:1261] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.475368 19410 consensus_queue.cc:1035] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:44.477104 16266 raft_consensus.cc:2932] T 88890b0913c0462bade8208dfa0f64b7 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:44.484293 19495 tablet_replica.cc:330] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: stopping tablet replica
I20240502 08:18:44.484566 19495 raft_consensus.cc:2229] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:44.484776 19495 pending_rounds.cc:62] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Trying to abort 1 pending ops.
I20240502 08:18:44.484892 19495 pending_rounds.cc:68] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023156935336853504 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "810c5f8a81ba4e0ab06129c0178f12e7" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } } }
I20240502 08:18:44.485453 19495 raft_consensus.cc:2894] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 08:18:44.485595 19495 raft_consensus.cc:2866] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:44.488796 19495 raft_consensus.cc:2258] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:44.491346 19495 ts_tablet_manager.cc:1875] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:44.504106 14062 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: config changed from index 11 to 12, 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } }
I20240502 08:18:44.506052 14059 catalog_manager.cc:5484] T 88890b0913c0462bade8208dfa0f64b7 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 7 to 8, NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } }
I20240502 08:18:44.519145 19383 tablet_copy_client.cc:783] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:44.519433 19383 tablet_copy_client.cc:647] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:44.522493 19383 tablet_copy_client.cc:523] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:18:44.527271 19383 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Bootstrap starting.
I20240502 08:18:44.538813 14336 consensus_queue.cc:237] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:44.545989 16400 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:44.546845 18963 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:44.551273 18963 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.555570 16400 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:44.563733 14064 catalog_manager.cc:5484] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:44.576057 16532 raft_consensus.cc:1261] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:44.576817 18963 consensus_queue.cc:1035] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:44.579324 16531 raft_consensus.cc:2932] T 6e901787c37a40e9a7c6db44091d11ef P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e79a5862de19490b89a9ea97860d776b (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
    @     0x7f3ead4ee8de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:18:44.613706 19111 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } }
I20240502 08:18:44.618693 16269 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } }
I20240502 08:18:44.621304 14336 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } }
I20240502 08:18:44.626088 14580 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e901787c37a40e9a7c6db44091d11ef with delete_type TABLET_DATA_TOMBSTONED (TS e79a5862de19490b89a9ea97860d776b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51824
I20240502 08:18:44.630388 18970 raft_consensus.cc:1050] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde: attempting to promote NON_VOTER f28bbf93da0c45eaaae0959e7a2f1e7e to VOTER
I20240502 08:18:44.631430 18970 consensus_queue.cc:237] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:44.635671 14059 catalog_manager.cc:5484] T 784f72f28102445f8c4c44a3e3da0d2d P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 7 to 8, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } }
I20240502 08:18:44.655953 19509 tablet_replica.cc:330] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: stopping tablet replica
I20240502 08:18:44.656299 19509 raft_consensus.cc:2229] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:44.656737 19509 raft_consensus.cc:2258] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:44.664602 19508 ts_tablet_manager.cc:902] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Initiating tablet copy from peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987)
I20240502 08:18:44.667624 19509 ts_tablet_manager.cc:1875] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:44.668568 16400 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.669885 18970 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:44.673950 14467 raft_consensus.cc:2932] T 784f72f28102445f8c4c44a3e3da0d2d P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } }
I20240502 08:18:44.674594 14598 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.675812 18970 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:44.688354 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e901787c37a40e9a7c6db44091d11ef with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:44.700559 14597 consensus_queue.cc:237] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:44.711338 14336 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:44.712031 19410 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:44.712541 19509 ts_tablet_manager.cc:1888] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:44.712710 19509 log.cc:1196] T 6e901787c37a40e9a7c6db44091d11ef P e79a5862de19490b89a9ea97860d776b: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/6e901787c37a40e9a7c6db44091d11ef
I20240502 08:18:44.715425 19410 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.719908 14336 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
W20240502 08:18:44.724094 14533 consensus_peers.cc:485] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930e3af93722422fb9237fd9b623ceac. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:44.729070 16137 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 LEARNER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:44.734061 19262 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
    @     0x7f3ead506daf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 08:18:44.846909 16513 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:44.923674 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:44.940596 19383 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:18:44.941316 19383 tablet_bootstrap.cc:492] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Bootstrap complete.
I20240502 08:18:44.965912 19383 ts_tablet_manager.cc:1367] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Time spent bootstrapping tablet: real 0.439s	user 0.106s	sys 0.081s
    @     0x7f3ead631480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 08:18:44.959676 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:44.748982 16399 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:44.967003 19383 raft_consensus.cc:348] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:44.967649 19383 raft_consensus.cc:729] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42116b63c31a436ca060c34691780d04, State: Initialized, Role: LEARNER
I20240502 08:18:44.967911 19383 consensus_queue.cc:260] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } }
I20240502 08:18:44.968641 19383 ts_tablet_manager.cc:1398] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:18:44.736635 14062 catalog_manager.cc:5484] T 930e3af93722422fb9237fd9b623ceac P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 9 to 10, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New cstate: current_term: 1 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:44.969599 14620 tablet_copy_service.cc:342] P e79a5862de19490b89a9ea97860d776b: Request end of tablet copy session 42116b63c31a436ca060c34691780d04-88890b0913c0462bade8208dfa0f64b7 received from {username='slave'} at 127.6.151.72:49117
I20240502 08:18:44.969760 14620 tablet_copy_service.cc:433] P e79a5862de19490b89a9ea97860d776b: ending tablet copy session 42116b63c31a436ca060c34691780d04-88890b0913c0462bade8208dfa0f64b7 on tablet 88890b0913c0462bade8208dfa0f64b7 with peer 42116b63c31a436ca060c34691780d04
I20240502 08:18:44.741243 18963 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:44.974259 19508 tablet_copy_client.cc:308] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:43987
I20240502 08:18:44.740136 14467 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:44.744055 14466 consensus_queue.cc:237] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } }
I20240502 08:18:44.743605 16137 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:44.744590 14469 raft_consensus.cc:1261] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:18:44.822245 14051 catalog_manager.cc:4830] TS e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): tablet 6e901787c37a40e9a7c6db44091d11ef (table rebalance_test_table_9 [id=9503bcd79ffe49ff885479bd0656e90b]) successfully deleted
W20240502 08:18:44.975822 14051 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.151.68:42587, user_credentials={real_user=slave}} blocked reactor thread for 153585us
I20240502 08:18:44.976074 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 930e3af93722422fb9237fd9b623ceac with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:44.867123 14336 consensus_queue.cc:237] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:44.753227 19495 ts_tablet_manager.cc:1888] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:44.979660 19495 log.cc:1196] T 810c5f8a81ba4e0ab06129c0178f12e7 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/810c5f8a81ba4e0ab06129c0178f12e7
I20240502 08:18:44.979564 19041 consensus_queue.cc:1035] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:44.831833 19407 raft_consensus.cc:1050] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: attempting to promote NON_VOTER 42116b63c31a436ca060c34691780d04 to VOTER
I20240502 08:18:44.981485 19407 consensus_queue.cc:237] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:44.983112 16400 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:44.984253 14226 tablet_copy_service.cc:140] P b19bedd6f7f94bed9c727f6c7210cc8a: Received BeginTabletCopySession request for tablet 7a92e7e903594d84a59f3ed058a5a3ec from peer e79a5862de19490b89a9ea97860d776b ({username='slave'} at 127.6.151.68:35325)
I20240502 08:18:44.984443 14226 tablet_copy_service.cc:161] P b19bedd6f7f94bed9c727f6c7210cc8a: Beginning new tablet copy session on tablet 7a92e7e903594d84a59f3ed058a5a3ec from peer e79a5862de19490b89a9ea97860d776b at {username='slave'} at 127.6.151.68:35325: session id = e79a5862de19490b89a9ea97860d776b-7a92e7e903594d84a59f3ed058a5a3ec
I20240502 08:18:44.989023 16134 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 08:18:44.993655 19195 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:18:44.995071 14206 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
    @     0x7f3ead669a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 08:18:44.995787 16249 tablet_service.cc:1514] Processing DeleteTablet for tablet 930e3af93722422fb9237fd9b623ceac with delete_type TABLET_DATA_TOMBSTONED (TS 5b84b46faddb47fcae86cc5f3f478bf1 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48900
I20240502 08:18:44.995074 14203 raft_consensus.cc:1261] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:44.998065 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:18:44.999094 16134 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P f28bbf93da0c45eaaae0959e7a2f1e7e [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
    @     0x7f3ead64c981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240502 08:18:45.002820 16531 raft_consensus.cc:1261] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 08:18:45.003311 16532 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:18:45.003737 19187 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:45.008132 14467 raft_consensus.cc:2932] T 930e3af93722422fb9237fd9b623ceac P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:45.014112 14226 tablet_copy_source_session.cc:205] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:18:45.015354 14051 catalog_manager.cc:4830] TS 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): tablet 810c5f8a81ba4e0ab06129c0178f12e7 (table rebalance_test_table_5 [id=379a9e9cff5c46a2a50cb3509b442cd9]) successfully deleted
    @     0x7f3ead64c70e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 08:18:45.021961 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:45.023485 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:18:45.023598 14598 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } }
I20240502 08:18:45.024188 19187 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:18:45.024788 19187 consensus_queue.cc:1035] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:18:45.025286 19187 consensus_queue.cc:1035] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
    @     0x7f3e9e0a5b75  kudu::ThreadPool::DispatchThread() at ??:0
I20240502 08:18:45.026558 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
    @     0x7f3e9e0a919c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f3e9e0a9019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 08:18:45.036075 14065 catalog_manager.cc:5484] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 7 to 8, f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
    @     0x7f3e9e08401e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f3ea14096db  start_thread at ??:0
    @     0x7f3e9a24771f  clone at ??:0
W20240502 08:18:45.051636 14400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.65:43987, user_credentials={real_user=slave}} blocked reactor thread for 46202.3us
W20240502 08:18:45.065503 14401 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 54854.5us
I20240502 08:18:45.067186 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:45.069415 19383 tablet_replica.cc:330] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: stopping tablet replica
I20240502 08:18:45.069622 19383 raft_consensus.cc:2229] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 08:18:45.069806 19383 pending_rounds.cc:62] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Trying to abort 1 pending ops.
I20240502 08:18:45.069923 19383 pending_rounds.cc:68] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023156938442313728 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "88890b0913c0462bade8208dfa0f64b7" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: true } } } }
I20240502 08:18:45.070365 19383 raft_consensus.cc:2894] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 08:18:45.070531 19383 raft_consensus.cc:2866] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:18:45.070679 19383 raft_consensus.cc:2258] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 08:18:45.071765 19383 ts_tablet_manager.cc:1875] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:45.079098 19546 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:45.083295 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84eaabbe91524d129a170ad3519c265a with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:45.087371 16531 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:45.096410 14062 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 10 to 11, VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:45.100134 19543 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:45.104301 16248 tablet_service.cc:1514] Processing DeleteTablet for tablet 84eaabbe91524d129a170ad3519c265a with delete_type TABLET_DATA_TOMBSTONED (TS 5b84b46faddb47fcae86cc5f3f478bf1 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:48900
I20240502 08:18:45.106715 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6272b2f3a9504e01832cb31f5b322980 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:45.109262 14206 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:45.115548 19557 ts_tablet_manager.cc:902] T 930e3af93722422fb9237fd9b623ceac P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
F20240502 08:18:45.116039 19557 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.queue_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.queue_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_cpu_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_cpu_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_wall_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_wall_time_us) (tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.queue_time_us, tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.queue_time_us) (tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_cpu_time_us, tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_cpu_time_us) (tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_wall_time_us, tablet-download-pool-4f109afd932d4c77a6b7cd63c417bc70.run_wall_time_us) (tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.queue_time_us, tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.queue_time_us) (tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_cpu_time_us, tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_cpu_time_us) (tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_wall_time_us, tablet-download-pool-5341f4bf3c9643859859ed0d9887cfca.run_wall_time_us) (tablet-download-pool-6272b2f3a9504e01832cb31f5b322980.queue_time_us, tablet-download-pool-6272b2f3a9504e01832cb31f5b322980.queue_time_us) (tablet-download-pool-6272b2f3a9504e01832cb31f5b322980.run_cpu_time_us, tablet-download-pool-6272b2f3a9504e01832cb31f5b322980.run_cpu_time_us) (tablet-download-pool-6272b2f3a9504e01832cb31f5b322980.run_wall_time_us, tablet-download-pool-6272b2f3a9504e01832cb31f5b322980.run_wall_time_us) (tablet-download-pool-6e901787c37a40e9a7c6db44091d11ef.queue_time_us, tablet-download-pool-6e901787c37a40e9a7c6db44091d11ef.queue_time_us) (tablet-download-pool-6e901787c37a40e9a7c6db44091d11ef.run_cpu_time_us, tablet-download-pool-6e901787c37a40e9a7c6db44091d11ef.run_cpu_time_us) (tablet-download-pool-6e901787c37a40e9a7c6db44091d11ef.run_wall_time_us, tablet-download-pool-6e901787c37a40e9a7c6db44091d11ef.run_wall_time_us) (tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.queue_time_us, tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.queue_time_us) (tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_cpu_time_us, tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_cpu_time_us) (tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_wall_time_us, tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_wall_time_us) (tablet-download-pool-810c5f8a81ba4e0ab06129c0178f12e7.queue_time_us, tablet-download-pool-810c5f8a81ba4e0ab06129c0178f12e7.queue_time_us) (tablet-download-pool-810c5f8a81ba4e0ab06129c0178f12e7.run_cpu_time_us, tablet-download-pool-810c5f8a81ba4e0ab06129c0178f12e7.run_cpu_time_us) (tablet-download-pool-810c5f8a81ba4e0ab06129c0178f12e7.run_wall_time_us, tablet-download-pool-810c5f8a81ba4e0ab06129c0178f12e7.run_wall_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.queue_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.queue_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_cpu_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_cpu_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_wall_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_wall_time_us) (tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.queue_time_us, tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.queue_time_us) (tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_cpu_time_us, tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_cpu_time_us) (tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_wall_time_us, tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_wall_time_us) (tablet-download-pool-930e3af93722422fb9237fd9b623ceac.queue_time_us, tablet-download-pool-930e3af93722422fb9237fd9b623ceac.queue_time_us) (tablet-download-pool-97d7768da26e4fff966e494446a278f2.queue_time_us, tablet-download-pool-97d7768da26e4fff966e494446a278f2.queue_time_us) (tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_cpu_time_us, tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_cpu_time_us) (tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_wall_time_us, tablet-download-pool-97d7768da26e4fff966e494446a278f2.run_wall_time_us) (tablet-download-pool-cf7124419bae40dfaaabdd2e673a3eb0.queue_time_us, tablet-download-pool-cf7124419bae40dfaaabdd2e673a3eb0.queue_time_us) (tablet-download-pool-cf7124419bae40dfaaabdd2e673a3eb0.run_cpu_time_us, tablet-download-pool-cf7124419bae40dfaaabdd2e673a3eb0.run_cpu_time_us) (tablet-download-pool-cf7124419bae40dfaaabdd2e673a3eb0.run_wall_time_us, tablet-download-pool-cf7124419bae40dfaaabdd2e673a3eb0.run_wall_time_us) (tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.queue_time_us, tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.queue_time_us) (tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_cpu_time_us, tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_cpu_time_us) (tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_wall_time_us, tablet-download-pool-d33a9164ef974a8caae83f3c22b87a51.run_wall_time_us) (tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.queue_time_us, tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.queue_time_us) (tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_cpu_time_us, tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_cpu_time_us) (tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_wall_time_us, tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_wall_time_us) (tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.queue_time_us, tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.queue_time_us) (tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_cpu_time_us, tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_cpu_time_us) (tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_wall_time_us, tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f98cb709dcd  google::LogMessage::Fail() at ??:0
    @     0x7f98cb70db93  google::LogMessage::SendToLog() at ??:0
    @     0x7f98cb7097cc  google::LogMessage::Flush() at ??:0
    @     0x7f98cb70af59  google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:18:45.130959 14469 consensus_queue.cc:237] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:45.134400 16531 raft_consensus.cc:1261] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:18:45.136900 19543 consensus_queue.cc:1035] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
    @     0x7f98ce44a742  kudu::TraceMetrics::InternName() at ??:0
I20240502 08:18:45.145869 19187 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:45.149128 19383 ts_tablet_manager.cc:1888] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:18:45.149296 19383 log.cc:1196] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-7/wal/wals/88890b0913c0462bade8208dfa0f64b7
I20240502 08:18:45.149853 19383 ts_tablet_manager.cc:902] T 88890b0913c0462bade8208dfa0f64b7 P 42116b63c31a436ca060c34691780d04: Initiating tablet copy from peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587)
I20240502 08:18:45.150843 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84eaabbe91524d129a170ad3519c265a with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:45.153177 16531 raft_consensus.cc:2932] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
    @     0x7f98ce416f8b  kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:18:45.160670 14062 catalog_manager.cc:5484] T 84eaabbe91524d129a170ad3519c265a P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 11 to 12, VOTER 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
    @     0x7f98ce410cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f98dd8638de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:18:45.177826 19508 ts_tablet_manager.cc:1875] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:18:45.180982 14065 catalog_manager.cc:5484] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a reported cstate change: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:45.187963 19543 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
    @     0x7f98dd87bdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 08:18:45.195197 16531 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
    @     0x7f98dd9a6480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 08:18:45.209231 14062 catalog_manager.cc:5484] T 4f109afd932d4c77a6b7cd63c417bc70 P 42116b63c31a436ca060c34691780d04 reported cstate change: config changed from index 12 to 13, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
I20240502 08:18:45.211455 14203 raft_consensus.cc:2932] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 42116b63c31a436ca060c34691780d04 (127.6.151.72) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } }
    @     0x7f98dd9dea8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f98dd9c1981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f98dd9c170e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 08:18:45.235914 19545 tablet_replica.cc:330] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: stopping tablet replica
I20240502 08:18:45.236150 19545 raft_consensus.cc:2229] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:45.236526 19545 raft_consensus.cc:2258] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
    @     0x7f98ce41ab75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f98ce41e19c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f98ce41e019  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
I20240502 08:18:45.309302 19508 ts_tablet_manager.cc:1888] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
W20240502 08:18:45.245975 14270 consensus_peers.cc:485] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0815524d0eb448438dae05f646f14580. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:45.334100 19508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a92e7e903594d84a59f3ed058a5a3ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:18:45.334057 14270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 88134.2us
I20240502 08:18:45.257202 14203 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:45.278389 16513 tablet_service.cc:1514] Processing DeleteTablet for tablet 6272b2f3a9504e01832cb31f5b322980 with delete_type TABLET_DATA_TOMBSTONED (TS 42116b63c31a436ca060c34691780d04 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:34882
    @     0x7f98ce3f901e  kudu::Thread::SuperviseThread() at ??:0
I20240502 08:18:45.335047 19556 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:45.336668 16512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
    @     0x7f98d177e6db  start_thread at ??:0
I20240502 08:18:45.338052 19508 tablet_copy_client.cc:783] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: tablet copy: Starting download of 0 data blocks...
I20240502 08:18:45.338325 19508 tablet_copy_client.cc:647] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:18:45.244076 16267 raft_consensus.cc:1261] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:18:45.247193 14317 tablet_service.cc:1514] Processing DeleteTablet for tablet 84eaabbe91524d129a170ad3519c265a with delete_type TABLET_DATA_TOMBSTONED (TS 8ae1cf63e5bd46cf9a4ba89651971dde not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:46340
I20240502 08:18:45.248651 14315 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:45.340740 19508 tablet_copy_client.cc:523] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
    @     0x7f98ca5bc71f  clone at ??:0
I20240502 08:18:45.281018 19545 ts_tablet_manager.cc:1875] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:45.355439 19508 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Bootstrap starting.
I20240502 08:18:45.360620 19556 consensus_queue.cc:1035] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:18:45.380995 19580 tablet_replica.cc:330] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: stopping tablet replica
I20240502 08:18:45.381999 19580 raft_consensus.cc:2229] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:45.383229 19580 raft_consensus.cc:2258] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:45.402812 14206 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:45.400992 18963 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:45.298979 14449 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:18:45.299237 14402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.71:39897, user_credentials={real_user=slave}} blocked reactor thread for 306741us
I20240502 08:18:45.326833 14447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:45.482709 16402 raft_consensus.cc:2932] T 6272b2f3a9504e01832cb31f5b322980 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { promote: false } } }
I20240502 08:18:45.499586 14065 catalog_manager.cc:5484] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:45.464517 19580 ts_tablet_manager.cc:1875] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:45.515473 14141 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.67:39913, user_credentials={real_user=slave}} blocked reactor thread for 45953.7us
I20240502 08:18:45.580844 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:45.607276 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:45.641070 14184 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:18:45.650082 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:45.661917 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:45.668843 16382 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:45.718987 14317 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:45.759145 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:18:45.777925 19580 ts_tablet_manager.cc:1888] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 08:18:45.896641 14469 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70"
dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41818
W20240502 08:18:45.953984 14140 connection.cc:535] client connection to 127.6.151.69:43257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:45.966174 14049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.151.72:44955 (error 108)
W20240502 08:18:46.015693 14140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:45.891780 19508 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:18:45.896565 14466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6"
dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41814
W20240502 08:18:45.965884 14139 connection.cc:535] client connection to 127.6.151.72:44955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:45.953536 14534 connection.cc:535] client connection to 127.6.151.69:43257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:46.016264 14466 raft_consensus.cc:595] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Received request to transfer leadership
W20240502 08:18:46.016352 14534 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:46.016659 19508 tablet_bootstrap.cc:492] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Bootstrap complete.
I20240502 08:18:45.429529 16267 raft_consensus.cc:2932] T 0815524d0eb448438dae05f646f14580 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
W20240502 08:18:45.964885 14533 connection.cc:535] client connection to 127.6.151.72:44955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:46.016993 19508 ts_tablet_manager.cc:1367] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Time spent bootstrapping tablet: real 0.662s	user 0.055s	sys 0.008s
I20240502 08:18:46.018029 19508 raft_consensus.cc:348] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } }
I20240502 08:18:46.018421 19508 raft_consensus.cc:729] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Initialized, Role: LEARNER
I20240502 08:18:45.780813 19580 log.cc:1196] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/84eaabbe91524d129a170ad3519c265a
I20240502 08:18:46.018719 19508 consensus_queue.cc:260] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: true } }
I20240502 08:18:46.019415 19508 ts_tablet_manager.cc:1398] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:18:45.896895 14469 raft_consensus.cc:595] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Received request to transfer leadership
W20240502 08:18:45.952992 14401 connection.cc:535] client connection to 127.6.151.69:43257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:45.409448 19545 ts_tablet_manager.cc:1888] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:18:45.768532 19619 raft_consensus.cc:482] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 32e074cd0dfd4311adc4e2a0f5d1ad8a)
W20240502 08:18:46.019766 14401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:45.953266 14271 connection.cc:535] client connection to 127.6.151.69:43257 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:45.965348 14270 connection.cc:535] client connection to 127.6.151.72:44955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:45.966413 14400 connection.cc:535] client connection to 127.6.151.72:44955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:46.020314 14271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:46.020969 14051 catalog_manager.cc:4830] TS 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): tablet 84eaabbe91524d129a170ad3519c265a (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:46.020347 19545 log.cc:1196] T 930e3af93722422fb9237fd9b623ceac P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/wals/930e3af93722422fb9237fd9b623ceac
W20240502 08:18:46.078701 14271 consensus_peers.cc:485] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.151350 14271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 90527.7us
W20240502 08:18:46.151551 14271 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.151641 14271 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.151741 14271 consensus_peers.cc:485] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.076193 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:46.152834 14533 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 76629.4us
W20240502 08:18:46.153002 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:46.153120 14533 consensus_peers.cc:485] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.153223 14533 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.153316 14533 consensus_peers.cc:485] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.067790 14401 consensus_peers.cc:485] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.153559 14401 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 85782.8us
W20240502 08:18:46.026415 14049 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 60315.4us
I20240502 08:18:46.025878 14226 tablet_copy_service.cc:342] P b19bedd6f7f94bed9c727f6c7210cc8a: Request end of tablet copy session e79a5862de19490b89a9ea97860d776b-7a92e7e903594d84a59f3ed058a5a3ec received from {username='slave'} at 127.6.151.68:35325
W20240502 08:18:46.078097 14139 consensus_peers.cc:485] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:46.154767 14226 tablet_copy_service.cc:433] P b19bedd6f7f94bed9c727f6c7210cc8a: ending tablet copy session e79a5862de19490b89a9ea97860d776b-7a92e7e903594d84a59f3ed058a5a3ec on tablet 7a92e7e903594d84a59f3ed058a5a3ec with peer e79a5862de19490b89a9ea97860d776b
W20240502 08:18:46.154815 14139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 76714.2us
W20240502 08:18:46.154987 14139 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.079443 14140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 63852us
I20240502 08:18:46.024098 19545 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48900 (request call id 1) took 1033 ms. Trace:
I20240502 08:18:46.155279 19545 rpcz_store.cc:276] 0502 08:18:44.990124 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 08:18:44.995660 (+  5536us) service_pool.cc:225] Handling call
0502 08:18:45.235704 (+240044us) ts_tablet_manager.cc:1120] Deleting tablet 930e3af93722422fb9237fd9b623ceac
0502 08:18:45.235719 (+    15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:18:45.374935 (+139216us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:45.409250 (+ 34315us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:46.024040 (+614790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":239743,"thread_start_us":241516,"threads_started":1}
W20240502 08:18:46.079277 14270 consensus_peers.cc:485] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.156708 14270 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 77435us
W20240502 08:18:46.156868 14270 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.156976 14270 consensus_peers.cc:462] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:46.157076 14270 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.083515 14534 consensus_peers.cc:485] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.157881 14534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 74363.8us
W20240502 08:18:46.158038 14534 consensus_peers.cc:485] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.158139 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:46.158236 19543 raft_consensus.cc:979] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: : Instructing follower 8ae1cf63e5bd46cf9a4ba89651971dde to start an election
I20240502 08:18:46.158398 19543 raft_consensus.cc:1067] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Signalling peer 8ae1cf63e5bd46cf9a4ba89651971dde to start an election
W20240502 08:18:46.079993 14400 consensus_peers.cc:485] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.158823 14400 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 78874us
W20240502 08:18:46.158996 14400 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:46.021675 19619 raft_consensus.cc:504] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
W20240502 08:18:46.160743 14140 consensus_peers.cc:485] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:46.161630 19619 leader_election.cc:288] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269), 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257), 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955)
I20240502 08:18:46.165100 14050 catalog_manager.cc:4830] TS 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): tablet 930e3af93722422fb9237fd9b623ceac (table rebalance_test_table_1 [id=a4fb3d570e7e4cb2aaa59d45ed8287ff]) successfully deleted
I20240502 08:18:46.258131 16582 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 08:18:23.289944 16588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:23.290817 16588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:23.331749 16588 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:27.227180 16588 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:18:29.937361 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:29.937572 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:29.943282 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:29.943470 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:29.944207 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:29.946475 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:29.946681 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:29.946833 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:29.946949 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:29.947084 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:29.947664 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:29.948341 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:29.951359 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:29.951902 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:30.163051 16588 rebalancer_tool.cc:1074] tablet 1017d794c61345959c52f062dd39ef35: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:30.293190 16588 rebalancer_tool.cc:1084] tablet 6272b2f3a9504e01832cb31f5b322980: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '42116b63c31a436ca060c34691780d04' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } }.
  Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: NON_VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
I20240502 08:18:30.293385 16588 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:31.562013 17750 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.6.151.126:35417, user_credentials={real_user=slave}} blocked reactor thread for 46077.8us
W20240502 08:18:32.378624 16712 kernel_stack_watchdog.cc:198] Thread 16588 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 354ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f718067d980  (unknown) at ??:0
    @           0x90ef77  __sanitizer::internal_sched_yield() at ??:?
    @           0x87462d  __interceptor_pthread_create at ??:?
    @     0x7f717d2ebd11  kudu::Thread::StartThread() at ??:0
    @           0xa4d187  kudu::Thread::Create() at ??:?
    @     0x7f717d30c7ab  kudu::ThreadPool::CreateThread() at ??:0
    @     0x7f717d3083ad  kudu::ThreadPool::DoSubmit() at ??:0
    @     0x7f717d30d4ff  kudu::ThreadPool::Submit() at ??:0
    @     0x7f71917ad774  kudu::tools::Ksck::FetchInfoFromTabletServers() at ??:0
    @     0x7f71917aed72  kudu::tools::Ksck::Run() at ??:0
    @     0x7f719208b07e  kudu::tools::RebalancerTool::RefreshKsckResults() at ??:0
    @     0x7f719207a652  kudu::tools::RebalancerTool::GetClusterRawInfo() at ??:0
    @     0x7f71920934e2  kudu::tools::RebalancerTool::AlgoBasedRunner::GetNextMovesImpl() at ??:0
    @     0x7f719208c729  kudu::tools::RebalancerTool::BaseRunner::GetNextMoves() at ??:0
    @     0x7f7192089563  kudu::tools::RebalancerTool::RunWith() at ??:0
    @     0x7f719208614c  kudu::tools::RebalancerTool::Run() at ??:0
W20240502 08:18:32.442286 16588 thread.cc:636] ksck-fetch [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.034s	sys 0.157s
W20240502 08:18:32.442468 16588 thread.cc:602] ksck-fetch [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.034s	sys 0.157s
W20240502 08:18:34.488190 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.488343 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.488489 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.488682 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.488785 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.488873 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.488961 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.489073 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.489173 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.489269 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.489367 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.489444 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.489543 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.489629 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.489782 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.489862 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.489953 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.490036 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.490129 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:34.736948 16588 rebalancer_tool.cc:1074] tablet 97d7768da26e4fff966e494446a278f2: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:35.041431 16588 rebalancer_tool.cc:1074] tablet fec3033140544f4994057d5b22973fff: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:35.501528 16588 rebalancer_tool.cc:1074] tablet 827a2fb0396441749a35977e2854bfb6: 'e79a5862de19490b89a9ea97860d776b' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:35.861191 16588 rebalancer_tool.cc:1074] tablet d33a9164ef974a8caae83f3c22b87a51: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:36.027654 16588 rebalancer_tool.cc:1084] tablet 88890b0913c0462bade8208dfa0f64b7: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '5b84b46faddb47fcae86cc5f3f478bf1' move ignored: Illegal state: tablet 88890b0913c0462bade8208dfa0f64b7: replica 5b84b46faddb47fcae86cc5f3f478bf1 is already present
I20240502 08:18:36.027812 16588 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:38.831399 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:38.831583 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:38.831795 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
I20240502 08:18:39.139837 16588 rebalancer_tool.cc:1074] tablet 76dcfec9be9f4f19a43d3f4603ae89d8: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '' move scheduled
I20240502 08:18:39.771579 16588 rebalancer_tool.cc:1074] tablet 2c1fecce883346459e21c29b98d95b7b: '5b84b46faddb47fcae86cc5f3f478bf1' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:40.021692 16588 rebalancer_tool.cc:1084] tablet cd647b6897bc4292a2dc70ef39abab5e: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Not found: Peer with uuid 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in consensus config
I20240502 08:18:40.021874 16588 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:42.690536 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b3dfd9d6472342689afd86203f7afe5f to server e79a5862de19490b89a9ea97860d776b
W20240502 08:18:42.690894 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:42.691125 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:42.691221 16588 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server f28bbf93da0c45eaaae0959e7a2f1e7e to server b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:42.773321 16588 rebalancer_tool.cc:1084] tablet f0bcdf713f124d46a22ddfc2873cd742: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }.
  Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:42.773448 16588 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:44.170670 19411 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableSchema -> {remote=127.6.151.126:35417, user_credentials={real_user=slave}} blocked reactor thread for 60708us
W20240502 08:18:44.495600 19411 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.6.151.126:35417, user_credentials={real_user=slave}} blocked reactor thread for 44841.2us
W20240502 08:18:45.961970 19630 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:45.947049 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.151.69:43257 (local address 127.0.0.1:35088)
0502 08:18:45.960458 (+ 13409us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:45.960585 (+   127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":13267,"threads_started":1}
Illegal state: tablet server f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): unacceptable health status UNAVAILABLE

W20240502 08:18:46.164413 14140 consensus_peers.cc:462] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:46.166920 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eaabbe91524d129a170ad3519c265a" candidate_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:46.197782 14335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84eaabbe91524d129a170ad3519c265a" candidate_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" is_pre_election: true
W20240502 08:18:46.309504 14140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 145080us
W20240502 08:18:46.309711 14140 consensus_peers.cc:485] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.174918 14401 consensus_peers.cc:485] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:46.309698 14335 raft_consensus.cc:1789] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: voting while tombstoned based on last-logged opid 1.10
W20240502 08:18:46.309820 14140 consensus_peers.cc:462] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:46.309883 14335 raft_consensus.cc:2454] T 84eaabbe91524d129a170ad3519c265a P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b84b46faddb47fcae86cc5f3f478bf1 in term 1.
W20240502 08:18:46.309895 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.309859 14401 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 134936us
W20240502 08:18:46.309998 14140 consensus_peers.cc:485] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:46.197963 14336 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6"
dest_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde"
 from {username='slave'} at 127.6.151.67:42857
I20240502 08:18:46.310222 14336 raft_consensus.cc:482] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20240502 08:18:46.310158 14401 consensus_peers.cc:485] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:46.310400 14336 raft_consensus.cc:3037] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:46.310515 14401 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.167402 16202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:46.311457 16202 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 144118us
W20240502 08:18:46.227237 14049 catalog_manager.cc:4573] TS 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6272b2f3a9504e01832cb31f5b322980: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:46.312024 14049 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 84892.5us
I20240502 08:18:46.313397 14336 raft_consensus.cc:504] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:46.176360 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:18:46.179639 19545 tablet_replica.cc:330] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: stopping tablet replica
W20240502 08:18:46.242053 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:46.315274 14533 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 73206.8us
W20240502 08:18:46.245393 16203 leader_election.cc:334] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:46.315690 16203 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 70384us
I20240502 08:18:46.315811 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:46.316035 14469 raft_consensus.cc:3032] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:46.316115 14469 raft_consensus.cc:729] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:46.316466 14469 consensus_queue.cc:260] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
I20240502 08:18:46.316960 14469 raft_consensus.cc:3037] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.318023 19545 raft_consensus.cc:2229] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:46.318461 19545 raft_consensus.cc:2258] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:46.319204 19545 ts_tablet_manager.cc:1875] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:18:46.319886 16202 leader_election.cc:334] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:46.321669 19543 raft_consensus.cc:979] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a: : Instructing follower b19bedd6f7f94bed9c727f6c7210cc8a to start an election
I20240502 08:18:46.321838 19543 raft_consensus.cc:1067] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Signalling peer b19bedd6f7f94bed9c727f6c7210cc8a to start an election
I20240502 08:18:46.321899 16204 leader_election.cc:302] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 5b84b46faddb47fcae86cc5f3f478bf1; no voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 42116b63c31a436ca060c34691780d04, f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:46.322180 16204 raft_consensus.cc:2670] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240502 08:18:46.322583 14203 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70"
dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
 from {username='slave'} at 127.6.151.67:59069
I20240502 08:18:46.322865 14203 raft_consensus.cc:482] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:46.322983 14203 raft_consensus.cc:3037] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.323314 16249 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 08:18:46.325595 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:46.329620 14203 raft_consensus.cc:504] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
W20240502 08:18:46.333209 14139 leader_election.cc:334] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:46.333482 14203 leader_election.cc:288] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 2 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257), 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955)
I20240502 08:18:46.334153 14466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: true dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:46.334406 14466 raft_consensus.cc:3032] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:46.334481 14466 raft_consensus.cc:729] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:46.334784 14466 consensus_queue.cc:260] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:46.335265 14466 raft_consensus.cc:3037] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.335696 14469 raft_consensus.cc:2454] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 2.
I20240502 08:18:46.336211 14336 leader_election.cc:288] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 2 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897), f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257)
I20240502 08:18:46.338327 14466 raft_consensus.cc:2454] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 2.
I20240502 08:18:46.339442 16400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a635dc6d64724295be4b4a7f68d0fad6" candidate_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b3dfd9d6472342689afd86203f7afe5f"
I20240502 08:18:46.339787 16400 raft_consensus.cc:3037] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.340456 19545 ts_tablet_manager.cc:1888] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 08:18:46.340577 19545 log.cc:1196] T 84eaabbe91524d129a170ad3519c265a P 5b84b46faddb47fcae86cc5f3f478bf1: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/wals/84eaabbe91524d129a170ad3519c265a
I20240502 08:18:46.341115 19545 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48900 (request call id 2) took 1237 ms. Trace:
I20240502 08:18:46.341207 19545 rpcz_store.cc:276] 0502 08:18:45.104086 (+     0us) service_pool.cc:168] Inserting onto call queue
0502 08:18:45.104147 (+    61us) service_pool.cc:225] Handling call
0502 08:18:46.156464 (+1052317us) ts_tablet_manager.cc:1120] Deleting tablet 84eaabbe91524d129a170ad3519c265a
0502 08:18:46.156476 (+    12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0502 08:18:46.179552 (+ 23076us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x616000170554. stack: 00007f89185e8756 00007f89185e8510 00007f891761b48d 0000000000955804 00000000009557cd 000000000095519a 00007f8923e5a12a 00007f8927bc31ab 00007f8927bf58d4 00007f8927bd6bc0 00007f8927bd694d 0000000000a60bcc 00007f891862eb74 00007f891863219b 00007f8918632018 0000000000a60bcc
0502 08:18:46.322553 (+143001us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:46.340305 (+ 17752us) tablet_metadata.cc:641] Metadata flushed
0502 08:18:46.341055 (+   750us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3045,"spinlock_wait_cycles":52054912,"tablet-delete.queue_time_us":1051962}
I20240502 08:18:46.345676 16400 raft_consensus.cc:2454] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ae1cf63e5bd46cf9a4ba89651971dde in term 2.
I20240502 08:18:46.346289 14272 leader_election.cc:302] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 8ae1cf63e5bd46cf9a4ba89651971dde, b3dfd9d6472342689afd86203f7afe5f; no voters: 
I20240502 08:18:46.346683 19676 raft_consensus.cc:2781] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:46.346715 14050 catalog_manager.cc:4830] TS 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): tablet 84eaabbe91524d129a170ad3519c265a (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:46.346834 19676 raft_consensus.cc:686] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 2 LEADER]: Becoming Leader. State: Replica: 8ae1cf63e5bd46cf9a4ba89651971dde, State: Running, Role: LEADER
I20240502 08:18:46.347178 19676 consensus_queue.cc:237] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } }
W20240502 08:18:46.347975 14271 leader_election.cc:334] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:46.368573 14140 leader_election.cc:334] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:46.368763 14140 leader_election.cc:302] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 42116b63c31a436ca060c34691780d04, f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:46.371434 19715 raft_consensus.cc:2726] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 08:18:46.381984 14598 raft_consensus.cc:1203] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
W20240502 08:18:46.401119 14270 consensus_peers.cc:462] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:46.401595 14062 catalog_manager.cc:5484] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: term changed from 1 to 2, leader changed from 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) to 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66). New cstate: current_term: 2 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:46.449923 14203 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1017d794c61345959c52f062dd39ef35"
dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:57756
I20240502 08:18:46.450218 14203 raft_consensus.cc:595] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Received request to transfer leadership
W20240502 08:18:46.584869 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:46.598460 19717 raft_consensus.cc:979] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: : Instructing follower e79a5862de19490b89a9ea97860d776b to start an election
I20240502 08:18:46.598731 19714 raft_consensus.cc:1067] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Signalling peer e79a5862de19490b89a9ea97860d776b to start an election
I20240502 08:18:46.601155 14598 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1017d794c61345959c52f062dd39ef35"
dest_uuid: "e79a5862de19490b89a9ea97860d776b"
 from {username='slave'} at 127.6.151.65:35503
I20240502 08:18:46.601450 14598 raft_consensus.cc:482] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:46.601593 14598 raft_consensus.cc:3037] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.604831 14598 raft_consensus.cc:504] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:46.606854 16269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017d794c61345959c52f062dd39ef35" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "5b84b46faddb47fcae86cc5f3f478bf1"
I20240502 08:18:46.606886 16400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017d794c61345959c52f062dd39ef35" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b3dfd9d6472342689afd86203f7afe5f"
I20240502 08:18:46.607082 16400 raft_consensus.cc:3037] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.607152 16269 raft_consensus.cc:3037] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.609907 16400 raft_consensus.cc:2454] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 2.
I20240502 08:18:46.610090 16269 raft_consensus.cc:2454] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 2.
I20240502 08:18:46.610587 14535 leader_election.cc:302] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5b84b46faddb47fcae86cc5f3f478bf1, b3dfd9d6472342689afd86203f7afe5f, e79a5862de19490b89a9ea97860d776b; no voters: 
I20240502 08:18:46.614614 19671 raft_consensus.cc:2781] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:46.615866 19671 raft_consensus.cc:686] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 2 LEADER]: Becoming Leader. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:46.617532 14203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1017d794c61345959c52f062dd39ef35" candidate_uuid: "e79a5862de19490b89a9ea97860d776b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:46.822785 16581 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 08:18:23.161968 16590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:23.162551 16590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:23.175669 16590 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:27.310765 16590 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:18:30.050163 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.050436 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.052317 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.052460 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.052529 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.052604 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.052713 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.054939 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.056582 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.056732 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:30.056813 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.065537 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.065719 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.078135 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:30.078298 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.078421 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
I20240502 08:18:30.273751 16590 rebalancer_tool.cc:1084] tablet 96365c4348464db59463182aea8c1c23: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '42116b63c31a436ca060c34691780d04' move ignored: Not found: Peer with uuid b19bedd6f7f94bed9c727f6c7210cc8a not found in consensus config
I20240502 08:18:30.273990 16590 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:33.049049 18004 meta_cache.cc:1330] Time spent looking up entry by ID: real 0.177s	user 0.000s	sys 0.000s
W20240502 08:18:34.602752 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.603034 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.603173 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.603322 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.603433 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.603574 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.603725 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.603866 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W2024050I20240502 08:18:46.616230 19671 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:46.617776 14203 raft_consensus.cc:3032] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:46.905272 14203 raft_consensus.cc:729] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b19bedd6f7f94bed9c727f6c7210cc8a, State: Running, Role: LEADER
W20240502 08:18:46.757316 14271 consensus_peers.cc:485] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:46.905658 14271 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 148380us
I20240502 08:18:46.905730 14203 consensus_queue.cc:260] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:46.627664 14598 leader_election.cc:288] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [CANDIDATE]: Term 2 election: Requested vote from peers b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703), b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897)
I20240502 08:18:46.906257 14203 raft_consensus.cc:3037] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Advancing to term 2
2 08:18:34.604125 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.604318 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.604450 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:34.917071 16590 rebalancer_tool.cc:1074] tablet d33a9164ef974a8caae83f3c22b87a51: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:35.112102 16590 rebalancer_tool.cc:1074] tablet 88890b0913c0462bade8208dfa0f64b7: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:35.479614 16590 rebalancer_tool.cc:1074] tablet 97d7768da26e4fff966e494446a278f2: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:35.848178 16590 rebalancer_tool.cc:1084] tablet 827a2fb0396441749a35977e2854bfb6: 'e79a5862de19490b89a9ea97860d776b' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Illegal state: tablet 827a2fb0396441749a35977e2854bfb6: replica b3dfd9d6472342689afd86203f7afe5f is already present
I20240502 08:18:35.848316 16590 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:38.249357 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:38.249877 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:38.250070 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:38.250236 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:38.250360 16590 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
I20240502 08:18:38.381064 16590 rebalancer_tool.cc:1074] tablet a635dc6d64724295be4b4a7f68d0fad6: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '' move scheduled
I20240502 08:18:38.832036 16590 rebalancer_tool.cc:1074] tablet cf7124419bae40dfaaabdd2e673a3eb0: '42116b63c31a436ca060c34691780d04' -> 'e79a5862de19490b89a9ea97860d776b' move scheduled
I20240502 08:18:39.296869 16590 rebalancer_tool.cc:1074] tablet e6d3c011ad9546e3b69e23cbef587dbc: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:39.923830 16590 rebalancer_tool.cc:1074] tablet 52331b69903943cb9d74f5ef5787dd00: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:40.146183 16590 rebalancer_tool.cc:1074] tablet 2c1fecce883346459e21c29b98d95b7b: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:40.252224 16590 rebalancer_tool.cc:1074] tablet 6379601e2b2d4bdabde2773fcbe06fb4: 'e79a5862de19490b89a9ea97860d776b' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:40.592510 16590 rebalancer_tool.cc:1074] tablet b44cc3d7fc9944c7abf33b4da92b2f6f: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '' move scheduled
I20240502 08:18:40.890910 16590 rebalancer_tool.cc:1074] tablet 84eaabbe91524d129a170ad3519c265a: '5b84b46faddb47fcae86cc5f3f478bf1' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:41.282195 16590 rebalancer_tool.cc:1074] tablet 784f72f28102445f8c4c44a3e3da0d2d: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '8ae1cf63e5bd46cf9a4ba89651971dde' move scheduled
I20240502 08:18:41.643255 16590 rebalancer_tool.cc:1074] tablet 930e3af93722422fb9237fd9b623ceac: 'e79a5862de19490b89a9ea97860d776b' -> '8ae1cf63e5bd46cf9a4ba89651971dde' move scheduled
I20240502 08:18:41.817464 16590 rebalancer_tool.cc:1074] tablet 810c5f8a81ba4e0ab06129c0178f12e7: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:42.207145 16590 rebalancer_tool.cc:1084] tablet f0bcdf713f124d46a22ddfc2873cd742: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '' move ignored: Not found: Peer with uuid 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in consensus config
I20240502 08:18:42.207338 16590 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:43.160434 19293 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.ConnectToMaster -> {remote=127.6.151.126:35417, user_credentials={real_user=slave}} blocked reactor thread for 50267.1us
W20240502 08:18:45.955567 19564 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:45.503945 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.151.69:43257 (local address 127.0.0.1:35066)
0502 08:18:45.625576 (+121631us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:45.625596 (+    20us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:45.625707 (+   111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:45.954102 (+328395us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121528}
W20240502 08:18:45.966378 19569 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:45.263404 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.151.72:44955 (local address 127.0.0.1:39166)
0502 08:18:45.306823 (+ 43419us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:45.306843 (+    20us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:45.306971 (+   128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:45.319119 (+ 12148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 08:18:45.319124 (+     5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 08:18:45.319169 (+    45us) client_negotiation.cc:189] Negotiated authn=SASL
0502 08:18:45.319855 (+   686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 08:18:45.319868 (+    13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 08:18:45.965974 (+646106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: BlockingRecv error: recv got EOF from 127.6.151.72:44955 (error 108)
Metrics: {"client-negotiator.queue_time_us":43302,"thread_start_us":53453,"threads_started":1}
Illegal state: tablet server f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): unacceptable health status UNAVAILABLE

W20240502 08:18:46.754348 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:46.907269 14533 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.6.151.72:44955, user_credentials={real_user=slave}} blocked reactor thread for 152924us
I20240502 08:18:46.888129 14597 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8"
dest_uuid: "e79a5862de19490b89a9ea97860d776b"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:47438
I20240502 08:18:46.908612 14597 raft_consensus.cc:595] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:46.909060 14203 raft_consensus.cc:2454] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e79a5862de19490b89a9ea97860d776b in term 2.
W20240502 08:18:46.642518 14140 consensus_peers.cc:462] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:46.909340 14140 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.6.151.69:43257, user_credentials={real_user=slave}} blocked reactor thread for 266782us
I20240502 08:18:46.866394 16584 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 08:18:22.736754 16587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:22.737233 16587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:22.755235 16587 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:26.325956 16587 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:18:28.186524 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.186877 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:28.187041 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.187162 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:28.187268 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.187386 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:28.187485 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:28.187588 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.187695 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:28.187815 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:28.187952 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:28.188071 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:28.188228 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.188340 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:28.324846 16587 rebalancer_tool.cc:1084] tablet d33a9164ef974a8caae83f3c22b87a51: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 08:18:28.325066 16587 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:30.280467 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.280673 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.280864 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.281035 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.281149 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:30.281260 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:30.281500 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.281685 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.281813 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.282008 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.282140 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.282277 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.282425 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:30.282624 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:30.302968 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.303108 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:30.303225 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:30.303333 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:30.303448 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:30.639716 16587 rebalancer_tool.cc:1074] tablet 930e3af93722422fb9237fd9b623ceac: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:30.905438 16587 rebalancer_tool.cc:1074] tablet f0bcdf713f124d46a22ddfc2873cd742: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:31.095690 16587 rebalancer_tool.cc:1074] tablet b44cc3d7fc9944c7abf33b4da92b2f6f: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:31.326786 16587 rebalancer_tool.cc:1074] tablet 7a92e7e903594d84a59f3ed058a5a3ec: 'e79a5862de19490b89a9ea97860d776b' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:31.570451 16587 rebalancer_tool.cc:1074] tablet 0b15694b0b284ecebd899522dc69d422: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:32.374838 16587 rebalancer_tool.cc:1074] tablet e6d3c011ad9546e3b69e23cbef587dbc: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:32.782014 16587 rebalancer_tool.cc:1084] tablet 2c1fecce883346459e21c29b98d95b7b: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Not found: Peer with uuid 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in consensus config
I20240502 08:18:32.782225 16587 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:36.918893 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:36.919090 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:36.919477 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:36.919790 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 8ae1cf63e5bd46cf9a4ba89651971dde
W20240502 08:18:36.919905 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:36.930712 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:36.931033 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:36.931133 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server f28bbf93da0c45eaaae0959e7a2f1e7e to server e79a5862de19490b89a9ea97860d776b
W20240502 08:18:36.931289 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:36.931488 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:37.412950 16587 rebalancer_tool.cc:1074] tablet 4f109afd932d4c77a6b7cd63c417bc70: 'e79a5862de19490b89a9ea97860d776b' -> '' move scheduled
I20240502 08:18:37.658123 16587 rebalancer_tool.cc:1074] tablet cd647b6897bc4292a2dc70ef39abab5e: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:38.024820 16587 rebalancer_tool.cc:1084] tablet 88890b0913c0462bade8208dfa0f64b7: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Not found: Peer with uuid b19bedd6f7f94bed9c727f6c7210cc8a not found in consensus config
I20240502 08:18:38.024966 16587 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:40.227257 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b3dfd9d6472342689afd86203f7afe5f to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:40.227586 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:40.228204 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:40.228332 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:40.228593 16587 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server f28bbf93da0c45eaaae0959e7a2f1e7e to server e79a5862de19490b89a9ea97860d776b
I20240502 08:18:40.693148 16587 rebalancer_tool.cc:1074] tablet 97d7768da26e4fff966e494446a278f2: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '' move scheduled
I20240502 08:18:40.936167 16587 rebalancer_tool.cc:1074] tablet 6e901787c37a40e9a7c6db44091d11ef: 'e79a5862de19490b89a9ea97860d776b' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:41.148658 16587 rebalancer_tool.cc:1074] tablet 84eaabbe91524d129a170ad3519c265a: '5b84b46faddb47fcae86cc5f3f478bf1' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:41.527973 16587 rebalancer_tool.cc:1074] tablet 96365c4348464db59463182aea8c1c23: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:42.163141 16587 rebalancer_tool.cc:1074] tablet 6272b2f3a9504e01832cb31f5b322980: '42116b63c31a436ca060c34691780d04' -> 'b19bedd6f7f94bed9c727f6c7210cc8a' move scheduled
I20240502 08:18:42.551581 16587 rebalancer_tool.cc:1084] tablet 6379601e2b2d4bdabde2773fcbe06fb4: 'e79a5862de19490b89a9ea97860d776b' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Illegal state: tablet 6379601e2b2d4bdabde2773fcbe06fb4: replica b3dfd9d6472342689afd86203f7afe5f is already present
I20240502 08:18:42.551774 16587 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:45.957257 19589 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:45.612978 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.151.69:43257 (local address 127.0.0.1:35078)
0502 08:18:45.613199 (+   221us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:45.613218 (+    19us) client_negotiation.cc:174] Beginning negotiation
0502 08:18:45.613330 (+   112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:18:45.956095 (+342765us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":111}
Illegal state: tablet server f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): unacceptable health status UNAVAILABLE

W20240502 08:18:46.904224 14128 kernel_stack_watchdog.cc:198] Thread 14140 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 200ms:
Kernel stack:
(could not read kernel stack)
User stack:
    @     0x7f9b90ebc980  (unknown) at ??:0
    @     0x7f9b90eb69c6  __GI___pthread_rwlock_wrlock at ??:0
    @     0x7f9b8ae3c904  google::LogMessage::Flush() at ??:0
    @     0x7f9b8ae3ca11  google::LogMessage::~LogMessage() at ??:0
    @     0x7f9b9927aa03  kudu::consensus::Peer::ProcessTabletCopyResponse() at ??:0
    @     0x7f9b9927eca1  kudu::consensus::Peer::SendNextRequest()::$_2::operator()() at ??:0
    @     0x7f9b9927eaae  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f9b947684e0  kudu::rpc::OutboundCall::CallCallback() at ??:0
    @     0x7f9b94769365  kudu::rpc::OutboundCall::SetFailed() at ??:0
    @     0x7f9b946febb3  kudu::rpc::Connection::Shutdown() at ??:0
    @     0x7f9b94786e51  kudu::rpc::ReactorThread::DestroyConnection() at ??:0
    @     0x7f9b9478be2b  kudu::rpc::ReactorThread::CompleteConnectionNegotiation() at ??:0
    @     0x7f9b9471437d  kudu::rpc::NegotiationCompletedTask::Run() at ??:0
    @     0x7f9b9478569d  kudu::rpc::ReactorThread::AsyncHandler() at ??:0
    @     0x7f9b94797f45  ev::base<>::method_thunk<>() at ??:0
W20240502 08:18:46.925595 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:46.926518 19714 raft_consensus.cc:1050] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a: attempting to promote NON_VOTER e79a5862de19490b89a9ea97860d776b to VOTER
I20240502 08:18:46.927845 19714 consensus_queue.cc:237] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } }
W20240502 08:18:46.932385 14140 consensus_peers.cc:462] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:46.932180 14059 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b reported cstate change: term changed from 1 to 2, leader changed from b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) to e79a5862de19490b89a9ea97860d776b (127.6.151.68). New cstate: current_term: 2 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:18:46.936283 14597 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 LEARNER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:46.937322 14469 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:18:46.937964 19714 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:18:46.938549 19717 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:18:46.940284 14140 consensus_peers.cc:485] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:46.944900 14466 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 2 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:46.946002 19676 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:18:46.953666 14270 consensus_peers.cc:462] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:46.969235 16400 raft_consensus.cc:1261] T a635dc6d64724295be4b4a7f68d0fad6 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:46.970934 19676 consensus_queue.cc:1035] T a635dc6d64724295be4b4a7f68d0fad6 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:46.977046 19717 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:46.981621 19671 raft_consensus.cc:979] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b: : Instructing follower b3dfd9d6472342689afd86203f7afe5f to start an election
I20240502 08:18:46.981828 19671 raft_consensus.cc:1067] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Signalling peer b3dfd9d6472342689afd86203f7afe5f to start an election
I20240502 08:18:46.983450 16402 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8"
dest_uuid: "b3dfd9d6472342689afd86203f7afe5f"
 from {username='slave'} at 127.6.151.68:50817
I20240502 08:18:46.983822 16402 raft_consensus.cc:482] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:46.983448 14597 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:46.983974 16402 raft_consensus.cc:3037] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:46.990204 14062 catalog_manager.cc:5484] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 7 to 8, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:46.995276 16402 raft_consensus.cc:504] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:46.999635 14597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8" candidate_uuid: "b3dfd9d6472342689afd86203f7afe5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "e79a5862de19490b89a9ea97860d776b"
I20240502 08:18:46.999977 14597 raft_consensus.cc:3032] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:47.000078 14597 raft_consensus.cc:729] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e79a5862de19490b89a9ea97860d776b, State: Running, Role: LEADER
I20240502 08:18:47.000506 14597 consensus_queue.cc:260] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:47.000764 16402 leader_election.cc:288] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 2 election: Requested vote from peers e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587), 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955), 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703)
I20240502 08:18:47.001070 14597 raft_consensus.cc:3037] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:18:47.001107 16335 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:47.004884 14597 raft_consensus.cc:2454] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3dfd9d6472342689afd86203f7afe5f in term 2.
I20240502 08:18:47.025483 14469 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e79a5862de19490b89a9ea97860d776b (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:47.030998 14203 consensus_queue.cc:237] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } }
I20240502 08:18:47.033109 14597 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:47.034013 14469 raft_consensus.cc:1261] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Refusing update from remote peer b19bedd6f7f94bed9c727f6c7210cc8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:47.034579 19715 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:47.035208 19715 consensus_queue.cc:1035] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:47.038888 19714 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:47.043752 14049 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a92e7e903594d84a59f3ed058a5a3ec with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:47.043354 14597 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
I20240502 08:18:47.046286 14059 catalog_manager.cc:5484] T 7a92e7e903594d84a59f3ed058a5a3ec P b19bedd6f7f94bed9c727f6c7210cc8a reported cstate change: config changed from index 8 to 9, VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New cstate: current_term: 1 leader_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:47.061697 14469 raft_consensus.cc:2932] T 7a92e7e903594d84a59f3ed058a5a3ec P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { promote: false } } }
W20240502 08:18:47.063223 16335 leader_election.cc:334] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:47.074868 16269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76dcfec9be9f4f19a43d3f4603ae89d8" candidate_uuid: "b3dfd9d6472342689afd86203f7afe5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "5b84b46faddb47fcae86cc5f3f478bf1"
I20240502 08:18:47.075100 16269 raft_consensus.cc:3037] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:47.078150 16269 raft_consensus.cc:2454] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3dfd9d6472342689afd86203f7afe5f in term 2.
W20240502 08:18:47.086544 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:47.089147 16336 leader_election.cc:302] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5b84b46faddb47fcae86cc5f3f478bf1, b3dfd9d6472342689afd86203f7afe5f, e79a5862de19490b89a9ea97860d776b; no voters: 42116b63c31a436ca060c34691780d04
I20240502 08:18:47.089917 19835 raft_consensus.cc:2781] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:47.103920 19835 raft_consensus.cc:686] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [term 2 LEADER]: Becoming Leader. State: Replica: b3dfd9d6472342689afd86203f7afe5f, State: Running, Role: LEADER
I20240502 08:18:47.104523 19835 consensus_queue.cc:237] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:47.116756 14059 catalog_manager.cc:5484] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f reported cstate change: term changed from 1 to 2, leader changed from e79a5862de19490b89a9ea97860d776b (127.6.151.68) to b3dfd9d6472342689afd86203f7afe5f (127.6.151.71). New cstate: current_term: 2 leader_uuid: "b3dfd9d6472342689afd86203f7afe5f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 08:18:47.147776 14140 consensus_peers.cc:462] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:47.219856 14140 consensus_peers.cc:462] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:47.235256 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:47.255290 14050 catalog_manager.cc:4573] TS f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 7a92e7e903594d84a59f3ed058a5a3ec: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:47.262914 14469 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f"
dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:41882
I20240502 08:18:47.263211 14469 raft_consensus.cc:595] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Received request to transfer leadership
I20240502 08:18:47.286892 14469 consensus_queue.cc:237] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } }
W20240502 08:18:47.290338 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:47.298848 14335 raft_consensus.cc:1261] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Refusing update from remote peer 32e074cd0dfd4311adc4e2a0f5d1ad8a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:47.299556 19543 consensus_queue.cc:1035] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:18:47.300571 14401 consensus_peers.cc:485] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:47.301625 14400 consensus_peers.cc:485] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:47.305136 14335 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:47.304453 19950 raft_consensus.cc:2932] T fec3033140544f4994057d5b22973fff P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:47.311854 14064 catalog_manager.cc:5484] T fec3033140544f4994057d5b22973fff P 8ae1cf63e5bd46cf9a4ba89651971dde reported cstate change: config changed from index 8 to 9, NON_VOTER 42116b63c31a436ca060c34691780d04 (127.6.151.72) added. New cstate: current_term: 1 leader_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false replace: true } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: NON_VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: true } } }
I20240502 08:18:47.361864 16400 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:47.363425 19675 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:47.364784 16269 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:47.365422 18633 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:47.382195 14335 consensus_queue.cc:237] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:47.386476 16402 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:18:47.386843 14271 consensus_peers.cc:485] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96365c4348464db59463182aea8c1c23. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:47.387737 14597 raft_consensus.cc:1261] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Refusing update from remote peer 8ae1cf63e5bd46cf9a4ba89651971dde: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:18:47.387804 19681 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:18:47.388985 14271 consensus_peers.cc:485] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:47.389398 19681 consensus_queue.cc:1035] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:47.392793 19950 raft_consensus.cc:979] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a: : Instructing follower 5b84b46faddb47fcae86cc5f3f478bf1 to start an election
I20240502 08:18:47.393023 19543 raft_consensus.cc:1067] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Signalling peer 5b84b46faddb47fcae86cc5f3f478bf1 to start an election
I20240502 08:18:47.393906 16267 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f"
dest_uuid: "5b84b46faddb47fcae86cc5f3f478bf1"
 from {username='slave'} at 127.6.151.67:58489
I20240502 08:18:47.394205 16267 raft_consensus.cc:482] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:18:47.394351 16267 raft_consensus.cc:3037] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:47.397818 16267 raft_consensus.cc:504] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:47.403581 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f" candidate_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a"
I20240502 08:18:47.403874 14469 raft_consensus.cc:3032] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:18:47.403983 14469 raft_consensus.cc:729] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 32e074cd0dfd4311adc4e2a0f5d1ad8a, State: Running, Role: LEADER
I20240502 08:18:47.404362 14469 consensus_queue.cc:260] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:47.404872 14469 raft_consensus.cc:3037] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:47.412539 14469 raft_consensus.cc:2454] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b84b46faddb47fcae86cc5f3f478bf1 in term 2.
I20240502 08:18:47.415148 14203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f" candidate_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a"
I20240502 08:18:47.415479 14203 raft_consensus.cc:3037] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:47.419857 14203 raft_consensus.cc:2454] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b84b46faddb47fcae86cc5f3f478bf1 in term 2.
I20240502 08:18:47.421000 16202 leader_election.cc:302] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 5b84b46faddb47fcae86cc5f3f478bf1, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 
I20240502 08:18:47.421108 16267 leader_election.cc:288] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [CANDIDATE]: Term 2 election: Requested vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), b3dfd9d6472342689afd86203f7afe5f (127.6.151.71:39897)
I20240502 08:18:47.421504 19965 raft_consensus.cc:2781] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:18:47.427266 19965 raft_consensus.cc:686] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 LEADER]: Becoming Leader. State: Replica: 5b84b46faddb47fcae86cc5f3f478bf1, State: Running, Role: LEADER
I20240502 08:18:47.427330 19676 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:47.427934 19965 consensus_queue.cc:237] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:47.431550 14598 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:47.436535 16402 raft_consensus.cc:2932] T 96365c4348464db59463182aea8c1c23 P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:47.436022 14064 catalog_manager.cc:5484] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: term changed from 1 to 2, leader changed from 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67) to 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70). New cstate: current_term: 2 leader_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:18:47.443120 14065 catalog_manager.cc:5484] T 96365c4348464db59463182aea8c1c23 P e79a5862de19490b89a9ea97860d776b reported cstate change: config changed from index 8 to 9, NON_VOTER 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } } }
I20240502 08:18:47.447468 14447 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:47.453172 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44cc3d7fc9944c7abf33b4da92b2f6f" candidate_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b3dfd9d6472342689afd86203f7afe5f"
I20240502 08:18:47.453514 16402 raft_consensus.cc:3037] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:18:47.455509 14203 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:18:47.456995 19675 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:18:47.464557 16402 raft_consensus.cc:2454] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b84b46faddb47fcae86cc5f3f478bf1 in term 2.
I20240502 08:18:47.464461 14597 consensus_queue.cc:237] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } }
I20240502 08:18:47.467175 16400 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:47.467681 16267 raft_consensus.cc:1261] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Refusing update from remote peer e79a5862de19490b89a9ea97860d776b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:47.467887 19675 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:47.470750 19675 consensus_queue.cc:1035] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:47.471290 19675 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:47.475947 16267 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:47.481379 16400 raft_consensus.cc:2932] T 1017d794c61345959c52f062dd39ef35 P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:47.489116 14065 catalog_manager.cc:5484] T 1017d794c61345959c52f062dd39ef35 P 5b84b46faddb47fcae86cc5f3f478bf1 reported cstate change: config changed from index 8 to 10, VOTER b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65) evicted. New cstate: current_term: 2 leader_uuid: "e79a5862de19490b89a9ea97860d776b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } }
I20240502 08:18:47.491665 14051 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1017d794c61345959c52f062dd39ef35 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:18:47.497493 16249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:47.498623 14184 tablet_service.cc:1514] Processing DeleteTablet for tablet 1017d794c61345959c52f062dd39ef35 with delete_type TABLET_DATA_TOMBSTONED (TS b19bedd6f7f94bed9c727f6c7210cc8a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53474
I20240502 08:18:47.507624 14317 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:18:47.516121 19990 tablet_replica.cc:330] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: stopping tablet replica
I20240502 08:18:47.516408 19990 raft_consensus.cc:2229] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:18:47.518290 19990 raft_consensus.cc:2258] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:18:47.518251 16267 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P 5b84b46faddb47fcae86cc5f3f478bf1 [term 2 FOLLOWER]: Refusing update from remote peer b3dfd9d6472342689afd86203f7afe5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:18:47.519372 19990 ts_tablet_manager.cc:1875] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:18:47.519596 19835 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 08:18:47.525069 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:47.530236 19990 ts_tablet_manager.cc:1888] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:18:47.531349 19990 log.cc:1196] T 1017d794c61345959c52f062dd39ef35 P b19bedd6f7f94bed9c727f6c7210cc8a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/1017d794c61345959c52f062dd39ef35
I20240502 08:18:47.532579 14049 catalog_manager.cc:4830] TS b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): tablet 1017d794c61345959c52f062dd39ef35 (table rebalance_test_table_7 [id=420ddf822b774b50bb1452eabf489570]) successfully deleted
I20240502 08:18:47.572759 16382 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 08:18:47.577687 14270 consensus_peers.cc:462] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:47.596083 14184 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:18:47.600391 14597 raft_consensus.cc:1261] T 76dcfec9be9f4f19a43d3f4603ae89d8 P e79a5862de19490b89a9ea97860d776b [term 2 FOLLOWER]: Refusing update from remote peer b3dfd9d6472342689afd86203f7afe5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 08:18:47.601161 19835 consensus_queue.cc:1035] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 08:18:47.649740 14140 consensus_peers.cc:462] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:47.675298 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:18:47.700413 16335 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:47.712237 14140 consensus_peers.cc:462] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:47.723073 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:47.733417 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:47.824465 14203 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Refusing update from remote peer 5b84b46faddb47fcae86cc5f3f478bf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:47.825392 19965 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:47.839937 14447 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:18:47.859283 16249 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:47.875017 14469 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 2 FOLLOWER]: Refusing update from remote peer 5b84b46faddb47fcae86cc5f3f478bf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:47.876019 19965 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:47.881244 16382 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:18:47.898312 16400 raft_consensus.cc:1261] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Refusing update from remote peer 5b84b46faddb47fcae86cc5f3f478bf1: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:18:47.899025 19977 consensus_queue.cc:1035] T b44cc3d7fc9944c7abf33b4da92b2f6f P 5b84b46faddb47fcae86cc5f3f478bf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:18:47.907255 14317 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:18:47.913823 14580 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:18:47.969779 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:47.981972 20043 ts_tablet_manager.cc:902] T 96365c4348464db59463182aea8c1c23 P 5b84b46faddb47fcae86cc5f3f478bf1: Initiating tablet copy from peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269)
F20240502 08:18:47.982389 20043 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0815524d0eb448438dae05f646f14580.queue_time_us, tablet-download-pool-0815524d0eb448438dae05f646f14580.queue_time_us) (tablet-download-pool-0815524d0eb448438dae05f646f14580.run_cpu_time_us, tablet-download-pool-0815524d0eb448438dae05f646f14580.run_cpu_time_us) (tablet-download-pool-0815524d0eb448438dae05f646f14580.run_wall_time_us, tablet-download-pool-0815524d0eb448438dae05f646f14580.run_wall_time_us) (tablet-download-pool-0b15694b0b284ecebd899522dc69d422.queue_time_us, tablet-download-pool-0b15694b0b284ecebd899522dc69d422.queue_time_us) (tablet-download-pool-0b15694b0b284ecebd899522dc69d422.run_cpu_time_us, tablet-download-pool-0b15694b0b284ecebd899522dc69d422.run_cpu_time_us) (tablet-download-pool-0b15694b0b284ecebd899522dc69d422.run_wall_time_us, tablet-download-pool-0b15694b0b284ecebd899522dc69d422.run_wall_time_us) (tablet-download-pool-1017d794c61345959c52f062dd39ef35.queue_time_us, tablet-download-pool-1017d794c61345959c52f062dd39ef35.queue_time_us) (tablet-download-pool-1017d794c61345959c52f062dd39ef35.run_cpu_time_us, tablet-download-pool-1017d794c61345959c52f062dd39ef35.run_cpu_time_us) (tablet-download-pool-1017d794c61345959c52f062dd39ef35.run_wall_time_us, tablet-download-pool-1017d794c61345959c52f062dd39ef35.run_wall_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.queue_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.queue_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_cpu_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_cpu_time_us) (tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_wall_time_us, tablet-download-pool-2c1fecce883346459e21c29b98d95b7b.run_wall_time_us) (tablet-download-pool-44f747cd88e04d23951f351b1578bbb8.queue_time_us, tablet-download-pool-44f747cd88e04d23951f351b1578bbb8.queue_time_us) (tablet-download-pool-44f747cd88e04d23951f351b1578bbb8.run_cpu_time_us, tablet-download-pool-44f747cd88e04d23951f351b1578bbb8.run_cpu_time_us) (tablet-download-pool-44f747cd88e04d23951f351b1578bbb8.run_wall_time_us, tablet-download-pool-44f747cd88e04d23951f351b1578bbb8.run_wall_time_us) (tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.queue_time_us, tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.queue_time_us) (tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_cpu_time_us, tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_cpu_time_us) (tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_wall_time_us, tablet-download-pool-76dcfec9be9f4f19a43d3f4603ae89d8.run_wall_time_us) (tablet-download-pool-784f72f28102445f8c4c44a3e3da0d2d.queue_time_us, tablet-download-pool-784f72f28102445f8c4c44a3e3da0d2d.queue_time_us) (tablet-download-pool-784f72f28102445f8c4c44a3e3da0d2d.run_cpu_time_us, tablet-download-pool-784f72f28102445f8c4c44a3e3da0d2d.run_cpu_time_us) (tablet-download-pool-784f72f28102445f8c4c44a3e3da0d2d.run_wall_time_us, tablet-download-pool-784f72f28102445f8c4c44a3e3da0d2d.run_wall_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.queue_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.queue_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_cpu_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_cpu_time_us) (tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_wall_time_us, tablet-download-pool-84eaabbe91524d129a170ad3519c265a.run_wall_time_us) (tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.queue_time_us, tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.queue_time_us) (tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_cpu_time_us, tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_cpu_time_us) (tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_wall_time_us, tablet-download-pool-88890b0913c0462bade8208dfa0f64b7.run_wall_time_us) (tablet-download-pool-930e3af93722422fb9237fd9b623ceac.queue_time_us, tablet-download-pool-930e3af93722422fb9237fd9b623ceac.queue_time_us) (tablet-download-pool-930e3af93722422fb9237fd9b623ceac.run_cpu_time_us, tablet-download-pool-930e3af93722422fb9237fd9b623ceac.run_cpu_time_us) (tablet-download-pool-930e3af93722422fb9237fd9b623ceac.run_wall_time_us, tablet-download-pool-930e3af93722422fb9237fd9b623ceac.run_wall_time_us) (tablet-download-pool-96365c4348464db59463182aea8c1c23.queue_time_us, tablet-download-pool-96365c4348464db59463182aea8c1c23.queue_time_us) (tablet-download-pool-b44cc3d7fc9944c7abf33b4da92b2f6f.queue_time_us, tablet-download-pool-b44cc3d7fc9944c7abf33b4da92b2f6f.queue_time_us) (tablet-download-pool-b44cc3d7fc9944c7abf33b4da92b2f6f.run_cpu_time_us, tablet-download-pool-b44cc3d7fc9944c7abf33b4da92b2f6f.run_cpu_time_us) (tablet-download-pool-b44cc3d7fc9944c7abf33b4da92b2f6f.run_wall_time_us, tablet-download-pool-b44cc3d7fc9944c7abf33b4da92b2f6f.run_wall_time_us) (tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.queue_time_us, tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.queue_time_us) (tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_cpu_time_us, tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_cpu_time_us) (tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_wall_time_us, tablet-download-pool-cd647b6897bc4292a2dc70ef39abab5e.run_wall_time_us) (tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.queue_time_us, tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.queue_time_us) (tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_cpu_time_us, tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_cpu_time_us) (tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_wall_time_us, tablet-download-pool-e6d3c011ad9546e3b69e23cbef587dbc.run_wall_time_us) (tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.queue_time_us, tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.queue_time_us) (tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_cpu_time_us, tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_cpu_time_us) (tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_wall_time_us, tablet-download-pool-f0bcdf713f124d46a22ddfc2873cd742.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
    @     0x7f891591ddcd  google::LogMessage::Fail() at ??:0
I20240502 08:18:48.000065 14184 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
    @     0x7f8915921b93  google::LogMessage::SendToLog() at ??:0
    @     0x7f891591d7cc  google::LogMessage::Flush() at ??:0
    @     0x7f891591ef59  google::LogMessageFatal::~LogMessageFatal() at ??:0
    @     0x7f891865e742  kudu::TraceMetrics::InternName() at ??:0
    @     0x7f891862af8b  kudu::ThreadPool::ThreadPool() at ??:0
    @     0x7f8918624cd6  kudu::ThreadPoolBuilder::Build() at ??:0
    @     0x7f8927a778de  kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
W20240502 08:18:48.064908 14270 consensus_peers.cc:462] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
    @     0x7f8927a8fdaf  kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
    @     0x7f8927bba480  kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240502 08:18:48.081974 14534 consensus_peers.cc:485] T d33a9164ef974a8caae83f3c22b87a51 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
    @     0x7f8927bf2a8d  kudu::tserver::TabletCopyRunnable::Run() at ??:0
    @     0x7f8927bd5981  kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
    @     0x7f8927bd570e  std::_Function_handler<>::_M_invoke() at ??:0
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f891862eb75  kudu::ThreadPool::DispatchThread() at ??:0
    @     0x7f891863219c  kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
    @     0x7f8918632019  std::_Function_handler<>::_M_invoke() at ??:0
W20240502 08:18:48.103986 14271 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
    @           0xa60bcd  std::function<>::operator()() at ??:?
    @     0x7f891860d01e  kudu::Thread::SuperviseThread() at ??:0
    @     0x7f891b9926db  start_thread at ??:0
    @     0x7f89147d071f  clone at ??:0
I20240502 08:18:48.129912 19717 raft_consensus.cc:482] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:18:48.130108 19717 raft_consensus.cc:504] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "f28bbf93da0c45eaaae0959e7a2f1e7e" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 43257 } attrs { promote: false } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }
I20240502 08:18:48.131116 19717 leader_election.cc:288] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257), 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955)
I20240502 08:18:48.132345 14469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f109afd932d4c77a6b7cd63c417bc70" candidate_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" candidate_term: 3 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" is_pre_election: true
I20240502 08:18:48.132599 14469 raft_consensus.cc:2454] T 4f109afd932d4c77a6b7cd63c417bc70 P 32e074cd0dfd4311adc4e2a0f5d1ad8a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b19bedd6f7f94bed9c727f6c7210cc8a in term 2.
W20240502 08:18:48.132668 14140 leader_election.cc:334] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:48.139847 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.141767 14139 leader_election.cc:334] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:48.141942 14139 leader_election.cc:302] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, b19bedd6f7f94bed9c727f6c7210cc8a; no voters: 42116b63c31a436ca060c34691780d04, f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:48.143445 20057 raft_consensus.cc:2726] T 4f109afd932d4c77a6b7cd63c417bc70 P b19bedd6f7f94bed9c727f6c7210cc8a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:48.170522 16583 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 08:18:22.670241 16586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:22.670761 16586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:22.683367 16586 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:25.715008 16586 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:18:27.606118 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:27.606334 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:27.606439 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:27.606551 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:27.606721 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:27.607081 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:27.607247 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:27.607373 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:27.607481 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:27.607594 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:27.607707 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:27.607818 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:27.607904 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:27.608006 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:27.853134 16586 rebalancer_tool.cc:1074] tablet 96365c4348464db59463182aea8c1c23: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:28.305490 16586 rebalancer_tool.cc:1074] tablet d33a9164ef974a8caae83f3c22b87a51: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:28.684736 16586 rebalancer_tool.cc:1074] tablet 2c1fecce883346459e21c29b98d95b7b: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:28.999078 16586 rebalancer_tool.cc:1074] tablet cf7124419bae40dfaaabdd2e673a3eb0: 'e79a5862de19490b89a9ea97860d776b' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:29.409874 16586 rebalancer_tool.cc:1074] tablet 9ee2cdcfd53a469c803f846458dd083b: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:29.866335 16586 rebalancer_tool.cc:1074] tablet a635dc6d64724295be4b4a7f68d0fad6: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:30.115181 16586 rebalancer_tool.cc:1074] tablet cd647b6897bc4292a2dc70ef39abab5e: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:30.352265 16586 rebalancer_tool.cc:1074] tablet 6272b2f3a9504e01832cb31f5b322980: 'e79a5862de19490b89a9ea97860d776b' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:30.427326 16586 rebalancer_tool.cc:1074] tablet 5341f4bf3c9643859859ed0d9887cfca: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:30.599707 16586 rebalancer_tool.cc:1084] tablet 930e3af93722422fb9237fd9b623ceac: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } }.
  Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:30.599862 16586 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:34.710860 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.713511 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.713835 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.714030 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.714141 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.714229 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.714344 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.714435 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:34.714602 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.714740 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 42116b63c31a436ca060c34691780d04 to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.714888 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.714998 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.715106 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.715194 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:34.715309 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:34.715413 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.715530 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:34.715633 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:34.833329 16586 rebalancer_tool.cc:1074] tablet 76dcfec9be9f4f19a43d3f4603ae89d8: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:35.515143 16586 rebalancer_tool.cc:1074] tablet 6e901787c37a40e9a7c6db44091d11ef: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:36.026978 16586 rebalancer_tool.cc:1074] tablet 0815524d0eb448438dae05f646f14580: 'e79a5862de19490b89a9ea97860d776b' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:36.391422 16586 rebalancer_tool.cc:1074] tablet fec3033140544f4994057d5b22973fff: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:36.554538 16586 rebalancer_tool.cc:1084] tablet 97d7768da26e4fff966e494446a278f2: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move ignored: Illegal state: tablet 97d7768da26e4fff966e494446a278f2: replica f28bbf93da0c45eaaae0959e7a2f1e7e is already present
I20240502 08:18:36.554723 16586 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:38.950570 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:38.950850 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 8ae1cf63e5bd46cf9a4ba89651971dde
W20240502 08:18:38.951018 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:38.951107 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:38.951203 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:38.951314 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:38.951416 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 5b84b46faddb47fcae86cc5f3f478bf1 to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:38.951495 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server f28bbf93da0c45eaaae0959e7a2f1e7e
I20240502 08:18:39.579385 16586 rebalancer_tool.cc:1084] tablet f0bcdf713f124d46a22ddfc2873cd742: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '' move ignored: Not found: Peer with uuid 32e074cd0dfd4311adc4e2a0f5d1ad8a not found in consensus config
I20240502 08:18:39.579542 16586 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:41.547151 18829 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.6.151.70:45703, user_credentials={real_user=slave}} blocked reactor thread for 48219us
W20240502 08:18:42.499430 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server 5b84b46faddb47fcae86cc5f3f478bf1 to server 8ae1cf63e5bd46cf9a4ba89651971dde
W20240502 08:18:42.499593 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 5b84b46faddb47fcae86cc5f3f478bf1 to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:42.499702 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:42.499791 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:42.499962 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b3dfd9d6472342689afd86203f7afe5f to server e79a5862de19490b89a9ea97860d776b
W20240502 08:18:42.500048 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 5b84b46faddb47fcae86cc5f3f478bf1 to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:42.500178 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:42.500248 16586 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:42.849742 16586 rebalancer_tool.cc:1074] tablet f0bcdf713f124d46a22ddfc2873cd742: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '' move scheduled
I20240502 08:18:43.153126 16586 rebalancer_tool.cc:1074] tablet 4f109afd932d4c77a6b7cd63c417bc70: 'b3dfd9d6472342689afd86203f7afe5f' -> '' move scheduled
I20240502 08:18:43.291688 16586 rebalancer_tool.cc:1074] tablet 0b15694b0b284ecebd899522dc69d422: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:43.558758 16586 rebalancer_tool.cc:1074] tablet 88890b0913c0462bade8208dfa0f64b7: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:44.183940 16586 rebalancer_tool.cc:1074] tablet 7a92e7e903594d84a59f3ed058a5a3ec: 'f28bbf93da0c45eaaae0959e7a2f1e7e' -> 'e79a5862de19490b89a9ea97860d776b' move scheduled
I20240502 08:18:45.424226 16586 rebalancer_tool.cc:1136] tablet cf7124419bae40dfaaabdd2e673a3eb0: 'e79a5862de19490b89a9ea97860d776b' -> '42116b63c31a436ca060c34691780d04' move completed: Incomplete: tablet cf7124419bae40dfaaabdd2e673a3eb0, TS e79a5862de19490b89a9ea97860d776b -> TS 42116b63c31a436ca060c34691780d04 move failed, target replica disappeared
I20240502 08:18:45.615272 16586 rebalancer_tool.cc:1136] tablet 6e901787c37a40e9a7c6db44091d11ef: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '42116b63c31a436ca060c34691780d04' move completed: OK
I20240502 08:18:46.235117 16586 rebalancer_tool.cc:1136] tablet 9ee2cdcfd53a469c803f846458dd083b: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move completed: OK
I20240502 08:18:46.266631 16586 rebalancer_tool.cc:1136] tablet 2c1fecce883346459e21c29b98d95b7b: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '5b84b46faddb47fcae86cc5f3f478bf1' move completed: Incomplete: tablet 2c1fecce883346459e21c29b98d95b7b, TS 32e074cd0dfd4311adc4e2a0f5d1ad8a -> TS 5b84b46faddb47fcae86cc5f3f478bf1 move failed, target replica disappeared
I20240502 08:18:46.357383 16586 rebalancer_tool.cc:1136] tablet 76dcfec9be9f4f19a43d3f4603ae89d8: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'b3dfd9d6472342689afd86203f7afe5f' move completed: OK
I20240502 08:18:46.465575 16586 rebalancer_tool.cc:1136] tablet 96365c4348464db59463182aea8c1c23: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move completed: OK
I20240502 08:18:46.557337 16586 rebalancer_tool.cc:1136] tablet 4f109afd932d4c77a6b7cd63c417bc70: 'b3dfd9d6472342689afd86203f7afe5f' -> '' move completed: OK
I20240502 08:18:46.630790 16586 rebalancer_tool.cc:1136] tablet d33a9164ef974a8caae83f3c22b87a51: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move completed: OK
I20240502 08:18:46.693854 16586 rebalancer_tool.cc:1136] tablet 6272b2f3a9504e01832cb31f5b322980: 'e79a5862de19490b89a9ea97860d776b' -> '42116b63c31a436ca060c34691780d04' move completed: Incomplete: tablet 6272b2f3a9504e01832cb31f5b322980, TS e79a5862de19490b89a9ea97860d776b -> TS 42116b63c31a436ca060c34691780d04 move failed, target replica disappeared
I20240502 08:18:46.760649 16586 rebalancer_tool.cc:1136] tablet 5341f4bf3c9643859859ed0d9887cfca: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move completed: OK
I20240502 08:18:46.923848 16586 rebalancer_tool.cc:1136] tablet 0815524d0eb448438dae05f646f14580: 'e79a5862de19490b89a9ea97860d776b' -> '5b84b46faddb47fcae86cc5f3f478bf1' move completed: OK
I20240502 08:18:47.059084 16586 rebalancer_tool.cc:1136] tablet f0bcdf713f124d46a22ddfc2873cd742: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '' move completed: OK
I20240502 08:18:47.138172 16586 rebalancer_tool.cc:1136] tablet fec3033140544f4994057d5b22973fff: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move completed: Incomplete: tablet fec3033140544f4994057d5b22973fff, TS b19bedd6f7f94bed9c727f6c7210cc8a -> TS b3dfd9d6472342689afd86203f7afe5f move failed, target replica disappeared
I20240502 08:18:47.151237 16586 rebalancer_tool.cc:1136] tablet cd647b6897bc4292a2dc70ef39abab5e: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move completed: OK
I20240502 08:18:47.194835 16586 rebalancer_tool.cc:1084] tablet 810c5f8a81ba4e0ab06129c0178f12e7: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move ignored: Illegal state: tablet 810c5f8a81ba4e0ab06129c0178f12e7: replica b3dfd9d6472342689afd86203f7afe5f is already present
I20240502 08:18:47.195068 16586 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): unacceptable health status UNAVAILABLE

W20240502 08:18:48.210218 14140 consensus_peers.cc:462] T f0bcdf713f124d46a22ddfc2873cd742 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:48.272720 14533 consensus_peers.cc:485] T 810c5f8a81ba4e0ab06129c0178f12e7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.286669 14140 consensus_peers.cc:462] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
I20240502 08:18:48.321017 16580 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
       Statistic       |   Value
-----------------------+-----------
 Minimum Replica Count | 0
 Maximum Replica Count | 23
 Average Replica Count | 11.250000

Per-table replica distribution summary:
 Replica Skew |  Value
--------------+----------
 Minimum      | 3
 Maximum      | 3
 Average      | 3.000000


stderr: W20240502 08:18:22.737824 16589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:22.738194 16589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:22.751232 16589 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:18:26.480018 16589 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:18:28.441736 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:28.441927 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:28.442157 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.442503 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.524387 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:28.524592 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:28.524685 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:28.524786 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:28.528280 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:28.535094 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:28.535301 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:28.535441 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:28.535655 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 32e074cd0dfd4311adc4e2a0f5d1ad8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:28.535790 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:28.535892 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a4fb3d570e7e4cb2aaa59d45ed8287ff: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 42116b63c31a436ca060c34691780d04
I20240502 08:18:29.023584 16589 rebalancer_tool.cc:1074] tablet 6272b2f3a9504e01832cb31f5b322980: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:29.089779 16589 rebalancer_tool.cc:1074] tablet 2c1fecce883346459e21c29b98d95b7b: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:29.494652 16589 rebalancer_tool.cc:1074] tablet 44f747cd88e04d23951f351b1578bbb8: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:29.984601 16589 rebalancer_tool.cc:1074] tablet 5341f4bf3c9643859859ed0d9887cfca: 'e79a5862de19490b89a9ea97860d776b' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:30.127305 16589 rebalancer_tool.cc:1074] tablet cf7124419bae40dfaaabdd2e673a3eb0: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:30.483527 16589 rebalancer_tool.cc:1074] tablet 4f109afd932d4c77a6b7cd63c417bc70: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:30.720789 16589 rebalancer_tool.cc:1074] tablet 784f72f28102445f8c4c44a3e3da0d2d: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:30.899144 16589 rebalancer_tool.cc:1074] tablet 76dcfec9be9f4f19a43d3f4603ae89d8: 'e79a5862de19490b89a9ea97860d776b' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:31.104539 16589 rebalancer_tool.cc:1074] tablet 1017d794c61345959c52f062dd39ef35: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move scheduled
I20240502 08:18:31.441576 16589 rebalancer_tool.cc:1074] tablet 6379601e2b2d4bdabde2773fcbe06fb4: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:32.108764 16589 rebalancer_tool.cc:1074] tablet 810c5f8a81ba4e0ab06129c0178f12e7: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:33.448632 16589 rebalancer_tool.cc:1074] tablet 84eaabbe91524d129a170ad3519c265a: 'e79a5862de19490b89a9ea97860d776b' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:33.854816 16589 rebalancer_tool.cc:1084] tablet 9ee2cdcfd53a469c803f846458dd083b: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move ignored: Illegal state: tablet 9ee2cdcfd53a469c803f846458dd083b: replica f28bbf93da0c45eaaae0959e7a2f1e7e is already present
I20240502 08:18:33.854987 16589 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:37.780351 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:37.780558 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 5b84b46faddb47fcae86cc5f3f478bf1 to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:37.780759 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1f43ee8167df4e8dbbdcb01012aec36c: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:37.780997 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9f670750f860417b897070d52d84d0c2: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:37.781157 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:37.781265 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f5e582fa17054f269c52c3a87841b87c: could not find any suitable replica to move from server 8ae1cf63e5bd46cf9a4ba89651971dde to server f28bbf93da0c45eaaae0959e7a2f1e7e
W20240502 08:18:37.781405 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e34a7e4087de424b9cab81bfbc828397: could not find any suitable replica to move from server b19bedd6f7f94bed9c727f6c7210cc8a to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:37.837538 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server 5b84b46faddb47fcae86cc5f3f478bf1
I20240502 08:18:38.259372 16589 rebalancer_tool.cc:1074] tablet f0bcdf713f124d46a22ddfc2873cd742: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '' move scheduled
I20240502 08:18:38.834228 16589 rebalancer_tool.cc:1074] tablet a635dc6d64724295be4b4a7f68d0fad6: 'e79a5862de19490b89a9ea97860d776b' -> '' move scheduled
I20240502 08:18:38.989220 16589 rebalancer_tool.cc:1074] tablet b44cc3d7fc9944c7abf33b4da92b2f6f: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '' move scheduled
I20240502 08:18:39.048179 16589 rebalancer_tool.cc:1084] tablet e6d3c011ad9546e3b69e23cbef587dbc: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } }.
  Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ae1cf63e5bd46cf9a4ba89651971dde" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 44269 } attrs { replace: true } } peers { permanent_uuid: "e79a5862de19490b89a9ea97860d776b" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 42587 } } peers { permanent_uuid: "42116b63c31a436ca060c34691780d04" member_type: VOTER last_known_addr { host: "127.6.151.72" port: 44955 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: true } }
I20240502 08:18:39.048380 16589 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:18:39.590246 16589 thread.cc:636] rpc reactor (reactor) Time spent creating pthread: real 0.531s	user 0.128s	sys 0.197s
W20240502 08:18:39.590453 16589 thread.cc:602] rpc reactor (reactor) Time spent starting thread: real 0.532s	user 0.128s	sys 0.198s
W20240502 08:18:43.436991 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b3dfd9d6472342689afd86203f7afe5f to server 42116b63c31a436ca060c34691780d04
W20240502 08:18:43.437219 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 379a9e9cff5c46a2a50cb3509b442cd9: could not find any suitable replica to move from server 42116b63c31a436ca060c34691780d04 to server b3dfd9d6472342689afd86203f7afe5f
W20240502 08:18:43.437332 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3ebcd209cc5e4e148d4fb46dfb1008af: could not find any suitable replica to move from server 5b84b46faddb47fcae86cc5f3f478bf1 to server 8ae1cf63e5bd46cf9a4ba89651971dde
W20240502 08:18:43.463913 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 420ddf822b774b50bb1452eabf489570: could not find any suitable replica to move from server 5b84b46faddb47fcae86cc5f3f478bf1 to server b19bedd6f7f94bed9c727f6c7210cc8a
W20240502 08:18:43.464035 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 291881f82dc743a9954be49c6437402f: could not find any suitable replica to move from server b3dfd9d6472342689afd86203f7afe5f to server e79a5862de19490b89a9ea97860d776b
W20240502 08:18:43.464206 16589 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9503bcd79ffe49ff885479bd0656e90b: could not find any suitable replica to move from server e79a5862de19490b89a9ea97860d776b to server b3dfd9d6472342689afd86203f7afe5f
I20240502 08:18:44.096670 16589 rebalancer_tool.cc:1074] tablet 930e3af93722422fb9237fd9b623ceac: '5b84b46faddb47fcae86cc5f3f478bf1' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:44.487612 16589 rebalancer_tool.cc:1074] tablet 88890b0913c0462bade8208dfa0f64b7: '8ae1cf63e5bd46cf9a4ba89651971dde' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move scheduled
I20240502 08:18:45.467633 16589 rebalancer_tool.cc:1074] tablet 0815524d0eb448438dae05f646f14580: '5b84b46faddb47fcae86cc5f3f478bf1' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:46.200883 16589 rebalancer_tool.cc:1136] tablet 5341f4bf3c9643859859ed0d9887cfca: 'e79a5862de19490b89a9ea97860d776b' -> '42116b63c31a436ca060c34691780d04' move completed: OK
I20240502 08:18:46.564986 16589 rebalancer_tool.cc:1136] tablet cf7124419bae40dfaaabdd2e673a3eb0: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move completed: OK
I20240502 08:18:46.664898 16589 rebalancer_tool.cc:1136] tablet 44f747cd88e04d23951f351b1578bbb8: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move completed: OK
I20240502 08:18:46.761076 16589 rebalancer_tool.cc:1136] tablet a635dc6d64724295be4b4a7f68d0fad6: 'e79a5862de19490b89a9ea97860d776b' -> '' move completed: OK
I20240502 08:18:46.825839 16589 rebalancer_tool.cc:1123] tablet 6272b2f3a9504e01832cb31f5b322980: 'b19bedd6f7f94bed9c727f6c7210cc8a' -> 'b3dfd9d6472342689afd86203f7afe5f' move is abandoned: Illegal state: 6272b2f3a9504e01832cb31f5b322980: source replica b19bedd6f7f94bed9c727f6c7210cc8a does not have REPLACE attribute set
I20240502 08:18:46.842176 16589 rebalancer_tool.cc:1136] tablet 2c1fecce883346459e21c29b98d95b7b: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move completed: OK
I20240502 08:18:47.086508 16589 rebalancer_tool.cc:1123] tablet 784f72f28102445f8c4c44a3e3da0d2d: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move is abandoned: Illegal state: 784f72f28102445f8c4c44a3e3da0d2d: source replica 8ae1cf63e5bd46cf9a4ba89651971dde does not have REPLACE attribute set
I20240502 08:18:47.136116 16589 rebalancer_tool.cc:1136] tablet 6379601e2b2d4bdabde2773fcbe06fb4: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> 'f28bbf93da0c45eaaae0959e7a2f1e7e' move completed: OK
I20240502 08:18:47.213797 16589 rebalancer_tool.cc:1136] tablet 810c5f8a81ba4e0ab06129c0178f12e7: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '42116b63c31a436ca060c34691780d04' move completed: OK
I20240502 08:18:47.229662 16589 rebalancer_tool.cc:1136] tablet 84eaabbe91524d129a170ad3519c265a: 'e79a5862de19490b89a9ea97860d776b' -> '5b84b46faddb47fcae86cc5f3f478bf1' move completed: Incomplete: tablet 84eaabbe91524d129a170ad3519c265a, TS e79a5862de19490b89a9ea97860d776b -> TS 5b84b46faddb47fcae86cc5f3f478bf1 move failed, target replica disappeared
I20240502 08:18:47.243921 16589 rebalancer_tool.cc:1136] tablet f0bcdf713f124d46a22ddfc2873cd742: '32e074cd0dfd4311adc4e2a0f5d1ad8a' -> '' move completed: OK
I20240502 08:18:47.314447 16589 rebalancer_tool.cc:1074] tablet fec3033140544f4994057d5b22973fff: 'f28bbf93da0c45eaaae0959e7a2f1e7e' -> '42116b63c31a436ca060c34691780d04' move scheduled
I20240502 08:18:47.434666 16589 rebalancer_tool.cc:1074] tablet 96365c4348464db59463182aea8c1c23: '8ae1cf63e5bd46cf9a4ba89651971dde' -> '5b84b46faddb47fcae86cc5f3f478bf1' move scheduled
I20240502 08:18:47.552368 16589 rebalancer_tool.cc:1084] tablet 7a92e7e903594d84a59f3ed058a5a3ec: 'f28bbf93da0c45eaaae0959e7a2f1e7e' -> '5b84b46faddb47fcae86cc5f3f478bf1' move ignored: Not found: Peer with uuid f28bbf93da0c45eaaae0959e7a2f1e7e not found in consensus config
I20240502 08:18:47.552654 16589 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): unacceptable health status UNAVAILABLE

E20240502 08:18:48.322070  6749 external_mini_cluster.cc:943] Process with UUID f28bbf93da0c45eaaae0959e7a2f1e7e has crashed
E20240502 08:18:48.322176  6749 external_mini_cluster.cc:943] Process with UUID 42116b63c31a436ca060c34691780d04 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
  0
  num_crashes
    Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
  Actual: it does.
I20240502 08:18:48.323661  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14124
I20240502 08:18:48.333635 14246 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:18:48.334493 14270 consensus_peers.cc:485] T 97d7768da26e4fff966e494446a278f2 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.336072 14401 connection.cc:535] client connection to 127.6.151.70:45703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.336300 14534 connection.cc:535] client connection to 127.6.151.70:45703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.337803 19951 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:18:48.336748 (+     0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.151.70:45703 (local address 127.6.151.67:48349)
0502 08:18:48.336960 (+   212us) negotiation.cc:107] Waiting for socket to connect
0502 08:18:48.337035 (+    75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":123}
W20240502 08:18:48.338927 14401 consensus_peers.cc:485] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.339279 14534 consensus_peers.cc:485] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.339435 14139 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.340482 14271 consensus_peers.cc:462] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111)
W20240502 08:18:48.354748 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.371646 14271 consensus_peers.cc:462] T 96365c4348464db59463182aea8c1c23 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111)
W20240502 08:18:48.425154 14270 consensus_peers.cc:485] T 6e901787c37a40e9a7c6db44091d11ef P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.428287 14271 consensus_peers.cc:485] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.445668 14534 consensus_peers.cc:485] T 1017d794c61345959c52f062dd39ef35 P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.458776 14270 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.481081 14270 consensus_peers.cc:462] T 0815524d0eb448438dae05f646f14580 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.493229 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:48.493268 14271 consensus_peers.cc:485] T 5341f4bf3c9643859859ed0d9887cfca P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.517007 14533 consensus_peers.cc:485] T e6d3c011ad9546e3b69e23cbef587dbc P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.530458 14534 consensus_peers.cc:485] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.555467 14401 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.570508 14049 catalog_manager.cc:4573] TS 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 6272b2f3a9504e01832cb31f5b322980: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.591276 16336 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.618366 14400 consensus_peers.cc:485] T 84eaabbe91524d129a170ad3519c265a P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Couldn't send request to peer 42116b63c31a436ca060c34691780d04. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.628896 14140 connection.cc:535] client connection to 127.6.151.70:45703 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.632824 14140 consensus_peers.cc:485] T 0b15694b0b284ecebd899522dc69d422 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.633023 14140 consensus_peers.cc:485] T 784f72f28102445f8c4c44a3e3da0d2d P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.634307 14140 consensus_peers.cc:485] T 6379601e2b2d4bdabde2773fcbe06fb4 P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.634485 14140 consensus_peers.cc:485] T 2c1fecce883346459e21c29b98d95b7b P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.634608 14140 consensus_peers.cc:485] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.640947 14140 consensus_peers.cc:485] T cd647b6897bc4292a2dc70ef39abab5e P b19bedd6f7f94bed9c727f6c7210cc8a -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:48.643085  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14124
W20240502 08:18:48.682850 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:48.688782 14534 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Couldn't send request to peer 5b84b46faddb47fcae86cc5f3f478bf1. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.701601 14271 consensus_peers.cc:485] T 52331b69903943cb9d74f5ef5787dd00 P 8ae1cf63e5bd46cf9a4ba89651971dde -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Couldn't send request to peer f28bbf93da0c45eaaae0959e7a2f1e7e. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:18:48.706326 14402 connection.cc:535] server connection from 127.6.151.65:34957 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.706388 14400 connection.cc:535] client connection to 127.6.151.65:43987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.706480 16335 connection.cc:535] server connection from 127.6.151.65:49557 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:48.706951 14270 connection.cc:535] server connection from 127.6.151.65:60975 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:48.707340  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14255
W20240502 08:18:48.708068 14400 consensus_peers.cc:485] T 6272b2f3a9504e01832cb31f5b322980 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Couldn't send request to peer b19bedd6f7f94bed9c727f6c7210cc8a. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:43987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:48.708184 14400 consensus_peers.cc:485] T 44f747cd88e04d23951f351b1578bbb8 P 32e074cd0dfd4311adc4e2a0f5d1ad8a -> Peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Couldn't send request to peer b19bedd6f7f94bed9c727f6c7210cc8a. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:43987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:48.713766 14376 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:18:48.879817 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:48.983848  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14255
W20240502 08:18:49.023502 14401 connection.cc:535] server connection from 127.6.151.66:41887 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:49.023854 14050 connection.cc:535] server connection from 127.6.151.66:47089 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:18:49.024077 14535 connection.cc:535] server connection from 127.6.151.66:51175 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:49.024124  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14385
I20240502 08:18:49.030411 14506 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:18:49.101513 14534 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer f28bbf93da0c45eaaae0959e7a2f1e7e (127.6.151.69:43257): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.69:43257: connect: Connection refused (error 111)
W20240502 08:18:49.206802 14535 consensus_peers.cc:485] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): Couldn't send request to peer 8ae1cf63e5bd46cf9a4ba89651971dde. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:44269: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:49.219919 14535 consensus_peers.cc:485] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 8ae1cf63e5bd46cf9a4ba89651971dde (127.6.151.66:44269): Couldn't send request to peer 8ae1cf63e5bd46cf9a4ba89651971dde. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:44269: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:18:49.267400 14533 consensus_peers.cc:462] T 88890b0913c0462bade8208dfa0f64b7 P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
W20240502 08:18:49.301919 14533 consensus_peers.cc:462] T 930e3af93722422fb9237fd9b623ceac P e79a5862de19490b89a9ea97860d776b -> Peer 42116b63c31a436ca060c34691780d04 (127.6.151.72:44955): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.151.72:44955: connect: Connection refused (error 111)
I20240502 08:18:49.305784  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14385
W20240502 08:18:49.348984 14535 connection.cc:535] client connection to 127.6.151.67:39913 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:49.349702  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14517
W20240502 08:18:49.350342 14535 consensus_peers.cc:485] T 9ee2cdcfd53a469c803f846458dd083b P e79a5862de19490b89a9ea97860d776b -> Peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): Couldn't send request to peer 32e074cd0dfd4311adc4e2a0f5d1ad8a. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.67:39913: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:49.355998 14639 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:49.401188 20088 raft_consensus.cc:482] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5b84b46faddb47fcae86cc5f3f478bf1)
I20240502 08:18:49.401403 20088 raft_consensus.cc:504] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "32e074cd0dfd4311adc4e2a0f5d1ad8a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 39913 } attrs { replace: true } } peers { permanent_uuid: "b19bedd6f7f94bed9c727f6c7210cc8a" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 43987 } } peers { permanent_uuid: "b3dfd9d6472342689afd86203f7afe5f" member_type: VOTER last_known_addr { host: "127.6.151.71" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5b84b46faddb47fcae86cc5f3f478bf1" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 45703 } attrs { promote: false } }
I20240502 08:18:49.402483 20088 leader_election.cc:288] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913), b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987), 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703)
W20240502 08:18:49.404776 16337 leader_election.cc:334] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 32e074cd0dfd4311adc4e2a0f5d1ad8a (127.6.151.67:39913): Network error: Client connection negotiation failed: client connection to 127.6.151.67:39913: connect: Connection refused (error 111)
W20240502 08:18:49.406193 16336 leader_election.cc:334] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5b84b46faddb47fcae86cc5f3f478bf1 (127.6.151.70:45703): Network error: Client connection negotiation failed: client connection to 127.6.151.70:45703: connect: Connection refused (error 111)
I20240502 08:18:49.406399 16336 leader_election.cc:302] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b3dfd9d6472342689afd86203f7afe5f; no voters: 32e074cd0dfd4311adc4e2a0f5d1ad8a, 5b84b46faddb47fcae86cc5f3f478bf1
W20240502 08:18:49.406661 16335 leader_election.cc:334] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b19bedd6f7f94bed9c727f6c7210cc8a (127.6.151.65:43987): Network error: Client connection negotiation failed: client connection to 127.6.151.65:43987: connect: Connection refused (error 111)
I20240502 08:18:49.406739 20088 raft_consensus.cc:2726] T b44cc3d7fc9944c7abf33b4da92b2f6f P b3dfd9d6472342689afd86203f7afe5f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 08:18:49.618683  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14517
W20240502 08:18:49.662395 16335 connection.cc:535] server connection from 127.6.151.68:50817 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:18:49.663074  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 16318
W20240502 08:18:49.663550 16337 consensus_peers.cc:485] T 76dcfec9be9f4f19a43d3f4603ae89d8 P b3dfd9d6472342689afd86203f7afe5f -> Peer e79a5862de19490b89a9ea97860d776b (127.6.151.68:42587): Couldn't send request to peer e79a5862de19490b89a9ea97860d776b. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.68:42587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:18:49.668690 16440 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:49.865217  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 16318
I20240502 08:18:49.899910  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 14034
I20240502 08:18:49.901093 14092 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:50.151006  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 14034
2024-05-02T08:18:50Z chronyd exiting
I20240502 08:18:50.214078  6749 test_util.cc:175] -----------------------------------------------
I20240502 08:18:50.214185  6749 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714637790512854-6749-0
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00> (51944 ms)
[----------] 1 test from ConcurrentRebalancersTest (51944 ms total)

[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN      ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1
I20240502 08:18:50.216827  6749 test_util.cc:268] Using random seed: 1086307677
I20240502 08:18:50.218783  6749 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:18:50.218863  6749 ts_itest-base.cc:113] --------------
I20240502 08:18:50.218910  6749 ts_itest-base.cc:114] 4 tablet servers
I20240502 08:18:50.218966  6749 ts_itest-base.cc:115] 1 replicas per TS
I20240502 08:18:50.219025  6749 ts_itest-base.cc:116] --------------
2024-05-02T08:18:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:18:50Z Disabled control of system clock
I20240502 08:18:50.271311  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:38907
--webserver_interface=127.6.151.126
--webserver_port=0
--builtin_ntp_servers=127.6.151.84:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.151.126:38907
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 08:18:50.626386 20114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:50.626781 20114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:50.626890 20114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:50.634622 20114 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:50.634833 20114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:18:50.634896 20114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:50.634953 20114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:18:50.635002 20114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:18:50.644649 20114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.151.126:38907
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:38907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.151.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:50.646008 20114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:50.647778 20114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:50.659541 20119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:50.659811 20120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:50.660456 20122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:50.661101 20114 server_base.cc:1034] running on GCE node
I20240502 08:18:50.661990 20114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:50.663676 20114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:50.664906 20114 hybrid_clock.cc:648] HybridClock initialized: now 1714637930664849 us; error 52 us; skew 500 ppm
I20240502 08:18:50.665287 20114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:50.668421 20114 webserver.cc:460] Webserver started at http://127.6.151.126:35891/ using document root <none> and password file <none>
I20240502 08:18:50.669204 20114 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:50.669306 20114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:50.669592 20114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:50.675381 20114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9b70339729a74685861bd54114f0b213"
format_stamp: "Formatted at 2024-05-02 08:18:50 on dist-test-slave-x3qn"
I20240502 08:18:50.676242 20114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9b70339729a74685861bd54114f0b213"
format_stamp: "Formatted at 2024-05-02 08:18:50 on dist-test-slave-x3qn"
I20240502 08:18:50.682057 20114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20240502 08:18:50.685801 20128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:50.687239 20114 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 08:18:50.687397 20114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9b70339729a74685861bd54114f0b213"
format_stamp: "Formatted at 2024-05-02 08:18:50 on dist-test-slave-x3qn"
I20240502 08:18:50.687585 20114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:50.712297 20114 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:50.713171 20114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:50.713433 20114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:50.732614 20114 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.126:38907
I20240502 08:18:50.732637 20179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.126:38907 every 8 connection(s)
I20240502 08:18:50.734074 20114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:18:50.738090 20180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:50.744248  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 20114
I20240502 08:18:50.744468  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:18:50.748334 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213: Bootstrap starting.
I20240502 08:18:50.751539 20180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:50.752681 20180 log.cc:824] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:50.755491 20180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213: No bootstrap required, opened a new log
I20240502 08:18:50.762809 20180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b70339729a74685861bd54114f0b213" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 38907 } }
I20240502 08:18:50.763192 20180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:50.763303 20180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b70339729a74685861bd54114f0b213, State: Initialized, Role: FOLLOWER
I20240502 08:18:50.763934 20180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b70339729a74685861bd54114f0b213" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 38907 } }
I20240502 08:18:50.764146 20180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:50.764276 20180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:50.764431 20180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:50.766777 20180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b70339729a74685861bd54114f0b213" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 38907 } }
I20240502 08:18:50.767285 20180 leader_election.cc:302] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b70339729a74685861bd54114f0b213; no voters: 
I20240502 08:18:50.767874 20180 leader_election.cc:288] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:50.768028 20185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:50.768726 20185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [term 1 LEADER]: Becoming Leader. State: Replica: 9b70339729a74685861bd54114f0b213, State: Running, Role: LEADER
I20240502 08:18:50.769294 20185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b70339729a74685861bd54114f0b213" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 38907 } }
I20240502 08:18:50.770040 20180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:18:50.772496 20186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b70339729a74685861bd54114f0b213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b70339729a74685861bd54114f0b213" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 38907 } } }
I20240502 08:18:50.772469 20187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b70339729a74685861bd54114f0b213. Latest consensus state: current_term: 1 leader_uuid: "9b70339729a74685861bd54114f0b213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b70339729a74685861bd54114f0b213" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 38907 } } }
I20240502 08:18:50.772965 20186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [sys.catalog]: This master's current role is: LEADER
I20240502 08:18:50.772979 20187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213 [sys.catalog]: This master's current role is: LEADER
I20240502 08:18:50.775730 20191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:18:50.781595 20191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:18:50.791316 20191 catalog_manager.cc:1347] Generated new cluster ID: d01d091d04a44743ba2ae6e789f4cf00
I20240502 08:18:50.791441 20191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:18:50.813302 20191 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:18:50.814716 20191 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:18:50.829191 20191 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9b70339729a74685861bd54114f0b213: Generated new TSK 0
I20240502 08:18:50.830394 20191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:18:50.843909  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.65:0
--local_ip_for_outbound_sockets=127.6.151.65
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--builtin_ntp_servers=127.6.151.84:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:18:51.199059 20204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:51.199393 20204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:51.199537 20204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:51.207342 20204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:51.207497 20204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:51.207676 20204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.65
I20240502 08:18:51.217446 20204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.65
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:51.218916 20204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:51.220692 20204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:51.232995 20209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:51.233726 20210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:51.234393 20212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:51.236586 20204 server_base.cc:1034] running on GCE node
I20240502 08:18:51.237437 20204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:51.239187 20204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:51.240435 20204 hybrid_clock.cc:648] HybridClock initialized: now 1714637931240369 us; error 61 us; skew 500 ppm
I20240502 08:18:51.240777 20204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:51.243237 20204 webserver.cc:460] Webserver started at http://127.6.151.65:43087/ using document root <none> and password file <none>
I20240502 08:18:51.244050 20204 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:51.244174 20204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:51.244495 20204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:51.247282 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "21f970fbfa08497f81784d2053e77171"
format_stamp: "Formatted at 2024-05-02 08:18:51 on dist-test-slave-x3qn"
I20240502 08:18:51.248080 20204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "21f970fbfa08497f81784d2053e77171"
format_stamp: "Formatted at 2024-05-02 08:18:51 on dist-test-slave-x3qn"
I20240502 08:18:51.253515 20204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 08:18:51.257189 20218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:51.258860 20204 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:18:51.259030 20204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "21f970fbfa08497f81784d2053e77171"
format_stamp: "Formatted at 2024-05-02 08:18:51 on dist-test-slave-x3qn"
I20240502 08:18:51.259239 20204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:51.276621 20204 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:51.277487 20204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:51.277801 20204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:51.278779 20204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:51.280309 20204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:51.280397 20204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:51.280483 20204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:51.280540 20204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:51.313050 20204 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.65:32959
I20240502 08:18:51.313112 20332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.65:32959 every 8 connection(s)
I20240502 08:18:51.314417 20204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:18:51.319995  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 20204
I20240502 08:18:51.320220  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:18:51.323467  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.66:0
--local_ip_for_outbound_sockets=127.6.151.66
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--builtin_ntp_servers=127.6.151.84:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:51.328388 20333 heartbeater.cc:347] Connected to a master server at 127.6.151.126:38907
I20240502 08:18:51.328807 20333 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:51.329761 20333 heartbeater.cc:511] Master 127.6.151.126:38907 requested a full tablet report, sending...
I20240502 08:18:51.332212 20145 ts_manager.cc:194] Registered new tserver with Master: 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959)
I20240502 08:18:51.334395 20145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.65:45739
W20240502 08:18:51.670261 20337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:51.670580 20337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:51.670737 20337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:51.678622 20337 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:51.678808 20337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:51.678977 20337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.66
I20240502 08:18:51.688804 20337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.66
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:51.690205 20337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:51.692050 20337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:51.706485 20342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:51.707016 20343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:51.707705 20345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:51.708403 20337 server_base.cc:1034] running on GCE node
I20240502 08:18:51.709065 20337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:51.710130 20337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:51.711349 20337 hybrid_clock.cc:648] HybridClock initialized: now 1714637931711302 us; error 37 us; skew 500 ppm
I20240502 08:18:51.711674 20337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:51.714023 20337 webserver.cc:460] Webserver started at http://127.6.151.66:37117/ using document root <none> and password file <none>
I20240502 08:18:51.714818 20337 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:51.714941 20337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:51.715250 20337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:51.717972 20337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "54eba73ef0bb466ab4776d6ea0541b78"
format_stamp: "Formatted at 2024-05-02 08:18:51 on dist-test-slave-x3qn"
I20240502 08:18:51.718822 20337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "54eba73ef0bb466ab4776d6ea0541b78"
format_stamp: "Formatted at 2024-05-02 08:18:51 on dist-test-slave-x3qn"
I20240502 08:18:51.723882 20337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20240502 08:18:51.727288 20351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:51.728562 20337 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:18:51.728724 20337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "54eba73ef0bb466ab4776d6ea0541b78"
format_stamp: "Formatted at 2024-05-02 08:18:51 on dist-test-slave-x3qn"
I20240502 08:18:51.728925 20337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:51.758901 20337 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:51.759800 20337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:51.760080 20337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:51.761060 20337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:51.762586 20337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:51.762720 20337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20240502 08:18:51.762846 20337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:51.762935 20337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:51.798173 20337 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.66:45629
I20240502 08:18:51.798225 20464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.66:45629 every 8 connection(s)
I20240502 08:18:51.799559 20337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:18:51.807590  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 20337
I20240502 08:18:51.807821  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:18:51.811359  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:0
--local_ip_for_outbound_sockets=127.6.151.67
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--builtin_ntp_servers=127.6.151.84:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:51.813573 20465 heartbeater.cc:347] Connected to a master server at 127.6.151.126:38907
I20240502 08:18:51.813871 20465 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:51.814749 20465 heartbeater.cc:511] Master 127.6.151.126:38907 requested a full tablet report, sending...
I20240502 08:18:51.816257 20145 ts_manager.cc:194] Registered new tserver with Master: 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629)
I20240502 08:18:51.817399 20145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.66:51071
W20240502 08:18:52.156648 20468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:52.156965 20468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:52.157085 20468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:52.164887 20468 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:52.165023 20468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:52.165174 20468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:18:52.174852 20468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:52.176198 20468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:52.177991 20468 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:52.190841 20473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:52.191077 20476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:52.191629 20474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:52.192425 20468 server_base.cc:1034] running on GCE node
I20240502 08:18:52.193065 20468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:52.194083 20468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:52.195294 20468 hybrid_clock.cc:648] HybridClock initialized: now 1714637932195239 us; error 43 us; skew 500 ppm
I20240502 08:18:52.195611 20468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:52.198000 20468 webserver.cc:460] Webserver started at http://127.6.151.67:36395/ using document root <none> and password file <none>
I20240502 08:18:52.198823 20468 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:52.198945 20468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:52.199277 20468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:52.201941 20468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7c2550bd25424f3f82ad02367f52be2a"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:18:52.202778 20468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7c2550bd25424f3f82ad02367f52be2a"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:18:52.208055 20468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:18:52.211397 20482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:52.212694 20468 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:18:52.212847 20468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7c2550bd25424f3f82ad02367f52be2a"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:18:52.213042 20468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:52.226714 20468 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:52.227519 20468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:52.227787 20468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:52.228693 20468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:52.230463 20468 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:52.230609 20468 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:52.230785 20468 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:52.230898 20468 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:52.271435 20468 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:36255
I20240502 08:18:52.271514 20595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:36255 every 8 connection(s)
I20240502 08:18:52.272794 20468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:18:52.275413  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 20468
I20240502 08:18:52.275580  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:18:52.279125  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:0
--local_ip_for_outbound_sockets=127.6.151.68
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--builtin_ntp_servers=127.6.151.84:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:52.286809 20596 heartbeater.cc:347] Connected to a master server at 127.6.151.126:38907
I20240502 08:18:52.287135 20596 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:52.287998 20596 heartbeater.cc:511] Master 127.6.151.126:38907 requested a full tablet report, sending...
I20240502 08:18:52.289438 20145 ts_manager.cc:194] Registered new tserver with Master: 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67:36255)
I20240502 08:18:52.290362 20145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:34619
I20240502 08:18:52.338066 20333 heartbeater.cc:503] Master 127.6.151.126:38907 was elected leader, sending a full tablet report...
W20240502 08:18:52.616400 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:52.616706 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:52.616830 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:52.624397 20599 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:52.624523 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:52.624693 20599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:18:52.633752 20599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:38907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:52.635218 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:52.636950 20599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:52.649772 20604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:52.649869 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:52.650439 20607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:52.651185 20599 server_base.cc:1034] running on GCE node
I20240502 08:18:52.651804 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:52.652773 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:52.653987 20599 hybrid_clock.cc:648] HybridClock initialized: now 1714637932653923 us; error 53 us; skew 500 ppm
I20240502 08:18:52.654323 20599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:52.656782 20599 webserver.cc:460] Webserver started at http://127.6.151.68:44261/ using document root <none> and password file <none>
I20240502 08:18:52.657681 20599 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:52.657814 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:52.658128 20599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:18:52.660822 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f6f542b9bfba42028f61587b9d915542"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:18:52.661607 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f6f542b9bfba42028f61587b9d915542"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:18:52.667045 20599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240502 08:18:52.670554 20613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:52.672026 20599 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:18:52.672173 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f6f542b9bfba42028f61587b9d915542"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:18:52.672343 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:52.694447 20599 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:52.695334 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:52.695586 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:52.696524 20599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:52.698014 20599 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:52.698101 20599 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:52.698199 20599 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:52.698256 20599 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:52.729832 20599 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:34235
I20240502 08:18:52.729884 20726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:34235 every 8 connection(s)
I20240502 08:18:52.731179 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:18:52.741462  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 20599
I20240502 08:18:52.741647  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:18:52.745167 20727 heartbeater.cc:347] Connected to a master server at 127.6.151.126:38907
I20240502 08:18:52.745478 20727 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:52.746268 20727 heartbeater.cc:511] Master 127.6.151.126:38907 requested a full tablet report, sending...
I20240502 08:18:52.747612 20145 ts_manager.cc:194] Registered new tserver with Master: f6f542b9bfba42028f61587b9d915542 (127.6.151.68:34235)
I20240502 08:18:52.748364 20145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:45437
I20240502 08:18:52.757534  6749 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 08:18:52.773581 20145 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42780:
name: "TestTable"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
owner: "alice"
I20240502 08:18:52.802938 20267 tablet_service.cc:1467] Processing CreateTablet for tablet 711469afa64b4d4dbccaeb20a2e5f483 (DEFAULT_TABLE table=TestTable [id=8db3be422c3b4045b1568952325b4fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:52.804674 20267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711469afa64b4d4dbccaeb20a2e5f483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:52.816371 20741 tablet_bootstrap.cc:492] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Bootstrap starting.
I20240502 08:18:52.819562 20741 tablet_bootstrap.cc:654] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:52.820535 20741 log.cc:824] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:52.822876 20741 tablet_bootstrap.cc:492] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: No bootstrap required, opened a new log
I20240502 08:18:52.823127 20741 ts_tablet_manager.cc:1367] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 08:18:52.826001 20465 heartbeater.cc:503] Master 127.6.151.126:38907 was elected leader, sending a full tablet report...
I20240502 08:18:52.829437 20741 raft_consensus.cc:348] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:52.829738 20741 raft_consensus.cc:374] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:52.829818 20741 raft_consensus.cc:729] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Initialized, Role: FOLLOWER
I20240502 08:18:52.830395 20741 consensus_queue.cc:260] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:52.830629 20741 raft_consensus.cc:388] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:52.830827 20741 raft_consensus.cc:482] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:52.830978 20741 raft_consensus.cc:3037] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:52.832866 20741 raft_consensus.cc:504] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:52.833334 20741 leader_election.cc:302] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171; no voters: 
I20240502 08:18:52.833914 20741 leader_election.cc:288] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:52.834095 20743 raft_consensus.cc:2781] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:52.835505 20741 ts_tablet_manager.cc:1398] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
I20240502 08:18:52.836045 20743 raft_consensus.cc:686] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Leader. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:18:52.836493 20743 consensus_queue.cc:237] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:52.841508 20145 catalog_manager.cc:5484] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f970fbfa08497f81784d2053e77171 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:52.850742  6749 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 08:18:52.852676  6749 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 21f970fbfa08497f81784d2053e77171 to finish bootstrapping
I20240502 08:18:52.861150  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 54eba73ef0bb466ab4776d6ea0541b78 to finish bootstrapping
I20240502 08:18:52.870244  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7c2550bd25424f3f82ad02367f52be2a to finish bootstrapping
I20240502 08:18:52.878919  6749 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f6f542b9bfba42028f61587b9d915542 to finish bootstrapping
I20240502 08:18:52.887317  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 20468
I20240502 08:18:52.888270 20591 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:52.970135  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 20468
I20240502 08:18:52.986872  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 20599
I20240502 08:18:52.988013 20722 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:18:53.069819  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 20599
I20240502 08:18:56.837910 20145 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42780:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:56.849025 20267 tablet_service.cc:1467] Processing CreateTablet for tablet b9d36140219745119ae5d6f34d41c66b (DEFAULT_TABLE table=rebalance_test_table_0 [id=1dd7c764b67a4fe7bfba877c635c3701]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:56.849435 20267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9d36140219745119ae5d6f34d41c66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:56.854996 20760 tablet_bootstrap.cc:492] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171: Bootstrap starting.
I20240502 08:18:56.856665 20760 tablet_bootstrap.cc:654] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:56.857270 20398 tablet_service.cc:1467] Processing CreateTablet for tablet f55c77486e5c42f99f05b43f82269720 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1dd7c764b67a4fe7bfba877c635c3701]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:56.857331 20399 tablet_service.cc:1467] Processing CreateTablet for tablet 3f57a6d216ee4ee6918af6a887a597c3 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1dd7c764b67a4fe7bfba877c635c3701]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:56.859019 20398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55c77486e5c42f99f05b43f82269720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:56.859264 20760 tablet_bootstrap.cc:492] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171: No bootstrap required, opened a new log
I20240502 08:18:56.859405 20760 ts_tablet_manager.cc:1367] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 08:18:56.859622 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f57a6d216ee4ee6918af6a887a597c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:56.860388 20760 raft_consensus.cc:348] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:56.860599 20760 raft_consensus.cc:374] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:56.860710 20760 raft_consensus.cc:729] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Initialized, Role: FOLLOWER
I20240502 08:18:56.860981 20760 consensus_queue.cc:260] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:56.861187 20760 raft_consensus.cc:388] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:56.861282 20760 raft_consensus.cc:482] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:56.861377 20760 raft_consensus.cc:3037] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:56.863411 20760 raft_consensus.cc:504] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:56.863670 20760 leader_election.cc:302] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171; no voters: 
I20240502 08:18:56.864555 20760 leader_election.cc:288] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:56.865998 20762 raft_consensus.cc:2781] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:56.868515 20762 raft_consensus.cc:686] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Leader. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:18:56.869088 20762 consensus_queue.cc:237] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:56.875345 20760 ts_tablet_manager.cc:1398] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.006s
I20240502 08:18:56.875638 20766 tablet_bootstrap.cc:492] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78: Bootstrap starting.
I20240502 08:18:56.877240 20145 catalog_manager.cc:5484] T b9d36140219745119ae5d6f34d41c66b P 21f970fbfa08497f81784d2053e77171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f970fbfa08497f81784d2053e77171 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:56.878804 20766 tablet_bootstrap.cc:654] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:56.880139 20766 log.cc:824] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78: Log is configured to *not* fsync() on all Append() calls
I20240502 08:18:56.882846 20766 tablet_bootstrap.cc:492] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78: No bootstrap required, opened a new log
I20240502 08:18:56.883175 20766 ts_tablet_manager.cc:1367] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.003s
I20240502 08:18:56.889756 20766 raft_consensus.cc:348] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.890096 20766 raft_consensus.cc:374] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:56.890208 20766 raft_consensus.cc:729] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Initialized, Role: FOLLOWER
I20240502 08:18:56.890877 20766 consensus_queue.cc:260] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.891079 20766 raft_consensus.cc:388] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:56.891209 20766 raft_consensus.cc:482] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:56.891376 20766 raft_consensus.cc:3037] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:56.893370 20766 raft_consensus.cc:504] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.893905 20766 leader_election.cc:302] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78; no voters: 
I20240502 08:18:56.894608 20766 leader_election.cc:288] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:56.894779 20769 raft_consensus.cc:2781] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:56.895393 20769 raft_consensus.cc:686] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Leader. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:18:56.896018 20769 consensus_queue.cc:237] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.896610 20766 ts_tablet_manager.cc:1398] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.001s
I20240502 08:18:56.897007 20766 tablet_bootstrap.cc:492] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: Bootstrap starting.
I20240502 08:18:56.898617 20766 tablet_bootstrap.cc:654] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:56.901338 20145 catalog_manager.cc:5484] T 3f57a6d216ee4ee6918af6a887a597c3 P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:56.902766 20766 tablet_bootstrap.cc:492] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: No bootstrap required, opened a new log
I20240502 08:18:56.902899 20766 ts_tablet_manager.cc:1367] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: Time spent bootstrapping tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 08:18:56.903681 20766 raft_consensus.cc:348] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.903822 20766 raft_consensus.cc:374] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:56.903961 20766 raft_consensus.cc:729] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Initialized, Role: FOLLOWER
I20240502 08:18:56.904237 20766 consensus_queue.cc:260] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.904446 20766 raft_consensus.cc:388] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:56.904529 20766 raft_consensus.cc:482] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:56.904637 20766 raft_consensus.cc:3037] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:56.907236 20766 raft_consensus.cc:504] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.907491 20766 leader_election.cc:302] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78; no voters: 
I20240502 08:18:56.907675 20766 leader_election.cc:288] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:56.907747 20769 raft_consensus.cc:2781] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:56.908193 20766 ts_tablet_manager.cc:1398] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:18:56.908210 20769 raft_consensus.cc:686] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Leader. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:18:56.908435 20769 consensus_queue.cc:237] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:56.910549 20144 catalog_manager.cc:5484] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:57.439036 20144 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42780:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:57.447428 20267 tablet_service.cc:1467] Processing CreateTablet for tablet 3afda5dcfefa48d5b95a0172847ef938 (DEFAULT_TABLE table=rebalance_test_table_1 [id=45d6e535683449bfb87fccecd5fc1061]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:57.447506 20398 tablet_service.cc:1467] Processing CreateTablet for tablet 50581ab834944158a50bfc5a7c96b57c (DEFAULT_TABLE table=rebalance_test_table_1 [id=45d6e535683449bfb87fccecd5fc1061]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:57.447613 20266 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd32afde3564645803d716a70853fe1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=45d6e535683449bfb87fccecd5fc1061]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:57.447957 20398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50581ab834944158a50bfc5a7c96b57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:57.447993 20267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afda5dcfefa48d5b95a0172847ef938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:57.448303 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd32afde3564645803d716a70853fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:57.455787 20799 tablet_bootstrap.cc:492] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171: Bootstrap starting.
I20240502 08:18:57.455785 20800 tablet_bootstrap.cc:492] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: Bootstrap starting.
I20240502 08:18:57.457062 20800 tablet_bootstrap.cc:654] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:57.457127 20799 tablet_bootstrap.cc:654] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:57.459012 20799 tablet_bootstrap.cc:492] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171: No bootstrap required, opened a new log
I20240502 08:18:57.459012 20800 tablet_bootstrap.cc:492] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: No bootstrap required, opened a new log
I20240502 08:18:57.459163 20799 ts_tablet_manager.cc:1367] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:18:57.459177 20800 ts_tablet_manager.cc:1367] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: Time spent bootstrapping tablet: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:18:57.459928 20800 raft_consensus.cc:348] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:57.459928 20799 raft_consensus.cc:348] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.460115 20799 raft_consensus.cc:374] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:57.460115 20800 raft_consensus.cc:374] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:57.460215 20800 raft_consensus.cc:729] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Initialized, Role: FOLLOWER
I20240502 08:18:57.460223 20799 raft_consensus.cc:729] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Initialized, Role: FOLLOWER
I20240502 08:18:57.460456 20800 consensus_queue.cc:260] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:57.460467 20799 consensus_queue.cc:260] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.460651 20800 raft_consensus.cc:388] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:57.460657 20799 raft_consensus.cc:388] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:57.460743 20799 raft_consensus.cc:482] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:57.460743 20800 raft_consensus.cc:482] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:57.460857 20799 raft_consensus.cc:3037] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:57.460903 20800 raft_consensus.cc:3037] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:57.463655 20800 raft_consensus.cc:504] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:57.463683 20799 raft_consensus.cc:504] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.463923 20799 leader_election.cc:302] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171; no voters: 
I20240502 08:18:57.463920 20800 leader_election.cc:302] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78; no voters: 
I20240502 08:18:57.464215 20800 leader_election.cc:288] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:57.464313 20769 raft_consensus.cc:2781] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:57.464388 20799 leader_election.cc:288] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:57.464568 20803 raft_consensus.cc:2781] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:57.475486 20769 raft_consensus.cc:686] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Leader. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:18:57.476013 20769 consensus_queue.cc:237] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:57.481089 20799 ts_tablet_manager.cc:1398] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171: Time spent starting tablet: real 0.022s	user 0.011s	sys 0.009s
I20240502 08:18:57.481199 20800 ts_tablet_manager.cc:1398] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: Time spent starting tablet: real 0.022s	user 0.021s	sys 0.000s
I20240502 08:18:57.481185 20144 catalog_manager.cc:5484] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:57.481431 20799 tablet_bootstrap.cc:492] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: Bootstrap starting.
I20240502 08:18:57.480428 20803 raft_consensus.cc:686] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Leader. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:18:57.482604 20803 consensus_queue.cc:237] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.483043 20799 tablet_bootstrap.cc:654] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:57.485002 20145 catalog_manager.cc:5484] T 3afda5dcfefa48d5b95a0172847ef938 P 21f970fbfa08497f81784d2053e77171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f970fbfa08497f81784d2053e77171 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:57.491425 20799 tablet_bootstrap.cc:492] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: No bootstrap required, opened a new log
I20240502 08:18:57.491570 20799 ts_tablet_manager.cc:1367] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.004s
I20240502 08:18:57.492208 20799 raft_consensus.cc:348] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.492386 20799 raft_consensus.cc:374] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:57.492465 20799 raft_consensus.cc:729] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Initialized, Role: FOLLOWER
I20240502 08:18:57.492686 20799 consensus_queue.cc:260] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.492826 20799 raft_consensus.cc:388] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:57.492925 20799 raft_consensus.cc:482] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:57.493016 20799 raft_consensus.cc:3037] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:57.494848 20799 raft_consensus.cc:504] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.495055 20799 leader_election.cc:302] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171; no voters: 
I20240502 08:18:57.495275 20799 leader_election.cc:288] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:57.495338 20803 raft_consensus.cc:2781] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:57.495582 20803 raft_consensus.cc:686] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Leader. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:18:57.495839 20803 consensus_queue.cc:237] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:57.495942 20799 ts_tablet_manager.cc:1398] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.005s
I20240502 08:18:57.497773 20145 catalog_manager.cc:5484] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f970fbfa08497f81784d2053e77171 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:58.010773 20145 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42780:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:58.020025 20398 tablet_service.cc:1467] Processing CreateTablet for tablet 8cacb00fa8b245a6b8604f90d8f5b618 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9e518ae71494d64ad8aa253c5d932f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:58.020198 20399 tablet_service.cc:1467] Processing CreateTablet for tablet f874df79af4c4080bc97fc9ad002b72e (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9e518ae71494d64ad8aa253c5d932f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:58.020273 20267 tablet_service.cc:1467] Processing CreateTablet for tablet 3eac9a0467f34a3986a0b9cfc61f51e0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b9e518ae71494d64ad8aa253c5d932f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:58.020571 20398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cacb00fa8b245a6b8604f90d8f5b618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:58.020660 20267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eac9a0467f34a3986a0b9cfc61f51e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:58.020990 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f874df79af4c4080bc97fc9ad002b72e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:58.026937 20837 tablet_bootstrap.cc:492] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: Bootstrap starting.
I20240502 08:18:58.027817 20838 tablet_bootstrap.cc:492] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171: Bootstrap starting.
I20240502 08:18:58.028424 20837 tablet_bootstrap.cc:654] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:58.029281 20838 tablet_bootstrap.cc:654] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:58.030412 20837 tablet_bootstrap.cc:492] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: No bootstrap required, opened a new log
I20240502 08:18:58.030558 20837 ts_tablet_manager.cc:1367] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: Time spent bootstrapping tablet: real 0.004s	user 0.000s	sys 0.003s
I20240502 08:18:58.031337 20838 tablet_bootstrap.cc:492] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171: No bootstrap required, opened a new log
I20240502 08:18:58.031391 20837 raft_consensus.cc:348] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.031489 20838 ts_tablet_manager.cc:1367] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:18:58.031569 20837 raft_consensus.cc:374] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:58.031675 20837 raft_consensus.cc:729] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Initialized, Role: FOLLOWER
I20240502 08:18:58.031930 20837 consensus_queue.cc:260] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.032128 20837 raft_consensus.cc:388] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:58.032223 20837 raft_consensus.cc:482] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:58.032214 20838 raft_consensus.cc:348] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.032337 20837 raft_consensus.cc:3037] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:58.032395 20838 raft_consensus.cc:374] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:58.032488 20838 raft_consensus.cc:729] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Initialized, Role: FOLLOWER
I20240502 08:18:58.032727 20838 consensus_queue.cc:260] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.032924 20838 raft_consensus.cc:388] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:58.033008 20838 raft_consensus.cc:482] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:58.033108 20838 raft_consensus.cc:3037] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:58.035126 20837 raft_consensus.cc:504] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.035377 20837 leader_election.cc:302] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78; no voters: 
I20240502 08:18:58.035845 20837 leader_election.cc:288] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:58.035849 20838 raft_consensus.cc:504] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.035980 20841 raft_consensus.cc:2781] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:58.036080 20838 leader_election.cc:302] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171; no voters: 
I20240502 08:18:58.036491 20803 raft_consensus.cc:2781] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:58.036613 20837 ts_tablet_manager.cc:1398] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20240502 08:18:58.036885 20837 tablet_bootstrap.cc:492] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78: Bootstrap starting.
I20240502 08:18:58.039147 20837 tablet_bootstrap.cc:654] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:58.041911 20837 tablet_bootstrap.cc:492] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78: No bootstrap required, opened a new log
I20240502 08:18:58.042191 20837 ts_tablet_manager.cc:1367] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:18:58.042724 20838 leader_election.cc:288] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:58.042924 20803 raft_consensus.cc:686] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Leader. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:18:58.043138 20837 raft_consensus.cc:348] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.043387 20837 raft_consensus.cc:374] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:58.043516 20837 raft_consensus.cc:729] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Initialized, Role: FOLLOWER
I20240502 08:18:58.043478 20803 consensus_queue.cc:237] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.043800 20841 raft_consensus.cc:686] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Leader. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:18:58.043859 20837 consensus_queue.cc:260] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.044068 20837 raft_consensus.cc:388] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:58.044114 20841 consensus_queue.cc:237] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.044207 20837 raft_consensus.cc:482] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:58.044337 20837 raft_consensus.cc:3037] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:58.047940 20837 raft_consensus.cc:504] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.048240 20837 leader_election.cc:302] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78; no voters: 
I20240502 08:18:58.048499 20837 leader_election.cc:288] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:58.048652 20843 raft_consensus.cc:2781] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:58.049367 20837 ts_tablet_manager.cc:1398] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20240502 08:18:58.052618 20838 ts_tablet_manager.cc:1398] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20240502 08:18:58.052996 20145 catalog_manager.cc:5484] T 3eac9a0467f34a3986a0b9cfc61f51e0 P 21f970fbfa08497f81784d2053e77171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f970fbfa08497f81784d2053e77171 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:58.054177 20843 raft_consensus.cc:686] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Leader. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:18:58.054545 20843 consensus_queue.cc:237] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.056272 20144 catalog_manager.cc:5484] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:58.061084 20144 catalog_manager.cc:5484] T f874df79af4c4080bc97fc9ad002b72e P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:58.570480 20144 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42780:
name: "rebalance_test_table_3"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
I20240502 08:18:58.579104 20399 tablet_service.cc:1467] Processing CreateTablet for tablet 0790c9f2e68e48d7b9513cc6623a348b (DEFAULT_TABLE table=rebalance_test_table_3 [id=8928a9601899448ca215c57c7857f8e6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:58.579079 20267 tablet_service.cc:1467] Processing CreateTablet for tablet 00a2296b66cd40f8ae82e8ca2354c065 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8928a9601899448ca215c57c7857f8e6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:58.579283 20266 tablet_service.cc:1467] Processing CreateTablet for tablet 9fd1e33dc4b4408dac968aa1fa33409d (DEFAULT_TABLE table=rebalance_test_table_3 [id=8928a9601899448ca215c57c7857f8e6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:18:58.579532 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0790c9f2e68e48d7b9513cc6623a348b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:58.579556 20267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a2296b66cd40f8ae82e8ca2354c065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:58.579764 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd1e33dc4b4408dac968aa1fa33409d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:18:58.585352 20877 tablet_bootstrap.cc:492] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: Bootstrap starting.
I20240502 08:18:58.586520 20878 tablet_bootstrap.cc:492] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78: Bootstrap starting.
I20240502 08:18:58.587210 20877 tablet_bootstrap.cc:654] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:58.587873 20878 tablet_bootstrap.cc:654] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:58.588989 20877 tablet_bootstrap.cc:492] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: No bootstrap required, opened a new log
I20240502 08:18:58.589171 20877 ts_tablet_manager.cc:1367] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:18:58.589543 20878 tablet_bootstrap.cc:492] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78: No bootstrap required, opened a new log
I20240502 08:18:58.589689 20878 ts_tablet_manager.cc:1367] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78: Time spent bootstrapping tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:18:58.589900 20877 raft_consensus.cc:348] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.590096 20877 raft_consensus.cc:374] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:58.590250 20877 raft_consensus.cc:729] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Initialized, Role: FOLLOWER
I20240502 08:18:58.590425 20878 raft_consensus.cc:348] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.590588 20878 raft_consensus.cc:374] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:58.590528 20877 consensus_queue.cc:260] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.590709 20878 raft_consensus.cc:729] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Initialized, Role: FOLLOWER
I20240502 08:18:58.590734 20877 raft_consensus.cc:388] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:58.590801 20877 raft_consensus.cc:482] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:58.590960 20877 raft_consensus.cc:3037] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:58.590919 20878 consensus_queue.cc:260] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.591118 20878 raft_consensus.cc:388] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:58.591193 20878 raft_consensus.cc:482] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:58.591302 20878 raft_consensus.cc:3037] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:58.592881 20877 raft_consensus.cc:504] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.593099 20877 leader_election.cc:302] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171; no voters: 
I20240502 08:18:58.593261 20878 raft_consensus.cc:504] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.593524 20877 leader_election.cc:288] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:58.593500 20878 leader_election.cc:302] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78; no voters: 
I20240502 08:18:58.593681 20878 leader_election.cc:288] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:58.593694 20881 raft_consensus.cc:2781] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:58.594000 20843 raft_consensus.cc:2781] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:58.607300 20843 raft_consensus.cc:686] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Leader. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:18:58.607821 20843 consensus_queue.cc:237] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } }
I20240502 08:18:58.613296 20144 catalog_manager.cc:5484] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:58.622385 20881 raft_consensus.cc:686] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Leader. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:18:58.622485 20878 ts_tablet_manager.cc:1398] T 0790c9f2e68e48d7b9513cc6623a348b P 54eba73ef0bb466ab4776d6ea0541b78: Time spent starting tablet: real 0.033s	user 0.009s	sys 0.023s
I20240502 08:18:58.622808 20881 consensus_queue.cc:237] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.625623 20877 ts_tablet_manager.cc:1398] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: Time spent starting tablet: real 0.036s	user 0.017s	sys 0.017s
I20240502 08:18:58.626498 20877 tablet_bootstrap.cc:492] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171: Bootstrap starting.
I20240502 08:18:58.627902 20145 catalog_manager.cc:5484] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f970fbfa08497f81784d2053e77171 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:58.628538 20877 tablet_bootstrap.cc:654] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171: Neither blocks nor log segments found. Creating new log.
I20240502 08:18:58.630486 20877 tablet_bootstrap.cc:492] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171: No bootstrap required, opened a new log
I20240502 08:18:58.630698 20877 ts_tablet_manager.cc:1367] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:18:58.631510 20877 raft_consensus.cc:348] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.631755 20877 raft_consensus.cc:374] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:18:58.631848 20877 raft_consensus.cc:729] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Initialized, Role: FOLLOWER
I20240502 08:18:58.632103 20877 consensus_queue.cc:260] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.632251 20877 raft_consensus.cc:388] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:18:58.632335 20877 raft_consensus.cc:482] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:18:58.632429 20877 raft_consensus.cc:3037] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:18:58.634474 20877 raft_consensus.cc:504] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.634711 20877 leader_election.cc:302] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171; no voters: 
I20240502 08:18:58.634903 20877 leader_election.cc:288] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:18:58.635059 20882 raft_consensus.cc:2781] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:18:58.635237 20882 raft_consensus.cc:686] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Leader. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:18:58.635380 20877 ts_tablet_manager.cc:1398] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.001s
I20240502 08:18:58.635638 20882 consensus_queue.cc:237] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } }
I20240502 08:18:58.638198 20145 catalog_manager.cc:5484] T 9fd1e33dc4b4408dac968aa1fa33409d P 21f970fbfa08497f81784d2053e77171 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f970fbfa08497f81784d2053e77171 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } health_report { overall_health: HEALTHY } } }
I20240502 08:18:59.148762  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:36255
--local_ip_for_outbound_sockets=127.6.151.67
--tserver_master_addrs=127.6.151.126:38907
--webserver_port=36395
--webserver_interface=127.6.151.67
--builtin_ntp_servers=127.6.151.84:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:18:59.494328 20912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:59.494655 20912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:59.494813 20912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:59.502292 20912 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:59.502461 20912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:59.502625 20912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:18:59.511657 20912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:36255
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=36395
--tserver_master_addrs=127.6.151.126:38907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:59.513007 20912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:59.514739 20912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:18:59.527026 20918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:59.527333 20917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:18:59.527921 20920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:18:59.528523 20912 server_base.cc:1034] running on GCE node
I20240502 08:18:59.529130 20912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:18:59.530055 20912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:18:59.531271 20912 hybrid_clock.cc:648] HybridClock initialized: now 1714637939531198 us; error 52 us; skew 500 ppm
I20240502 08:18:59.531731 20912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:18:59.534190 20912 webserver.cc:460] Webserver started at http://127.6.151.67:36395/ using document root <none> and password file <none>
I20240502 08:18:59.535008 20912 fs_manager.cc:359] Metadata directory not provided
I20240502 08:18:59.535117 20912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:18:59.540372 20912 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:18:59.543102 20926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:59.544499 20912 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:18:59.544637 20912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7c2550bd25424f3f82ad02367f52be2a"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:18:59.545432 20912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:18:59.579984 20912 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:18:59.580792 20912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:18:59.581046 20912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:18:59.581950 20912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:18:59.583415 20912 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:18:59.583487 20912 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:59.583578 20912 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:18:59.583623 20912 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:18:59.615167 20912 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:36255
I20240502 08:18:59.615227 21038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:36255 every 8 connection(s)
I20240502 08:18:59.616655 20912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:18:59.620909  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 20912
I20240502 08:18:59.621817  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:34235
--local_ip_for_outbound_sockets=127.6.151.68
--tserver_master_addrs=127.6.151.126:38907
--webserver_port=44261
--webserver_interface=127.6.151.68
--builtin_ntp_servers=127.6.151.84:40921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:18:59.632318 21039 heartbeater.cc:347] Connected to a master server at 127.6.151.126:38907
I20240502 08:18:59.632645 21039 heartbeater.cc:464] Registering TS with master...
I20240502 08:18:59.633606 21039 heartbeater.cc:511] Master 127.6.151.126:38907 requested a full tablet report, sending...
I20240502 08:18:59.635197 20145 ts_manager.cc:194] Re-registered known tserver with Master: 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67:36255)
I20240502 08:18:59.636400 20145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:56705
W20240502 08:18:59.967247 21043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:18:59.967547 21043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:18:59.967648 21043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:18:59.975291 21043 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:18:59.975430 21043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:18:59.975551 21043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:18:59.984972 21043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:34235
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=44261
--tserver_master_addrs=127.6.151.126:38907
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:18:59.986282 21043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:18:59.988010 21043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:00.000205 21051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:00.000418 21048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:00.000931 21049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:00.001751 21043 server_base.cc:1034] running on GCE node
I20240502 08:19:00.002344 21043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:00.003294 21043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:00.004504 21043 hybrid_clock.cc:648] HybridClock initialized: now 1714637940004444 us; error 52 us; skew 500 ppm
I20240502 08:19:00.004830 21043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:00.007125 21043 webserver.cc:460] Webserver started at http://127.6.151.68:44261/ using document root <none> and password file <none>
I20240502 08:19:00.007851 21043 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:00.007974 21043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:00.013159 21043 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20240502 08:19:00.015897 21057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:00.017267 21043 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:19:00.017426 21043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f6f542b9bfba42028f61587b9d915542"
format_stamp: "Formatted at 2024-05-02 08:18:52 on dist-test-slave-x3qn"
I20240502 08:19:00.018283 21043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:00.040807 21043 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:00.041720 21043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:00.042021 21043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:00.043185 21043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:00.044776 21043 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:00.044934 21043 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:00.045110 21043 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:00.045173 21043 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:00.077410 21043 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:34235
I20240502 08:19:00.077445 21170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:34235 every 8 connection(s)
I20240502 08:19:00.078896 21043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:19:00.086916  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 21043
I20240502 08:19:00.091430 21171 heartbeater.cc:347] Connected to a master server at 127.6.151.126:38907
I20240502 08:19:00.091743 21171 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:00.092646 21171 heartbeater.cc:511] Master 127.6.151.126:38907 requested a full tablet report, sending...
I20240502 08:19:00.094125 20145 ts_manager.cc:194] Re-registered known tserver with Master: f6f542b9bfba42028f61587b9d915542 (127.6.151.68:34235)
I20240502 08:19:00.094946 20145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:44477
I20240502 08:19:00.586162 20266 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:19:00.587620 20399 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:19:00.591863 20974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:00.602370 21106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:00.639170 21039 heartbeater.cc:503] Master 127.6.151.126:38907 was elected leader, sending a full tablet report...
I20240502 08:19:00.791010 20974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:00.791946 20399 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:19:00.791976 21106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:00.792074 20266 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:19:00.975574 20974 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:00.976469 20266 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:19:00.990128 20399 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:19:00.999539 21106 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:01.065245 20419 consensus_queue.cc:237] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.067901 21290 raft_consensus.cc:2932] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER f6f542b9bfba42028f61587b9d915542 (127.6.151.68) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } } }
I20240502 08:19:01.074340 20145 catalog_manager.cc:5484] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: config changed from index -1 to 3, NON_VOTER f6f542b9bfba42028f61587b9d915542 (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:19:01.079500 20354 consensus_peers.cc:485] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 -> Peer f6f542b9bfba42028f61587b9d915542 (127.6.151.68:34235): Couldn't send request to peer f6f542b9bfba42028f61587b9d915542. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cacb00fa8b245a6b8604f90d8f5b618. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:01.097308 21171 heartbeater.cc:503] Master 127.6.151.126:38907 was elected leader, sending a full tablet report...
I20240502 08:19:01.098318 20287 consensus_queue.cc:237] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.101063 21303 raft_consensus.cc:2932] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } } }
I20240502 08:19:01.107059 20145 catalog_manager.cc:5484] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 reported cstate change: config changed from index -1 to 2, NON_VOTER 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) added. New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:19:01.111394 20221 consensus_peers.cc:485] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 -> Peer 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67:36255): Couldn't send request to peer 7c2550bd25424f3f82ad02367f52be2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7fd32afde3564645803d716a70853fe1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:01.119462 20419 consensus_queue.cc:237] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.121191 21290 raft_consensus.cc:2932] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f6f542b9bfba42028f61587b9d915542 (127.6.151.68) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } } }
W20240502 08:19:01.121707 20354 consensus_peers.cc:485] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 -> Peer f6f542b9bfba42028f61587b9d915542 (127.6.151.68:34235): Couldn't send request to peer f6f542b9bfba42028f61587b9d915542. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50581ab834944158a50bfc5a7c96b57c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:01.126396 20141 catalog_manager.cc:5484] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: config changed from index -1 to 2, NON_VOTER f6f542b9bfba42028f61587b9d915542 (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:01.136502 20287 consensus_queue.cc:237] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.138048 21302 raft_consensus.cc:2932] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } } }
W20240502 08:19:01.138520 20221 consensus_peers.cc:485] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 -> Peer 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67:36255): Couldn't send request to peer 7c2550bd25424f3f82ad02367f52be2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 711469afa64b4d4dbccaeb20a2e5f483. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:01.142859 20141 catalog_manager.cc:5484] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 reported cstate change: config changed from index -1 to 2, NON_VOTER 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) added. New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:01.154153 20419 consensus_queue.cc:237] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.156262 21290 raft_consensus.cc:2932] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) added. New config: { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } } }
I20240502 08:19:01.161440 20141 catalog_manager.cc:5484] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: config changed from index -1 to 2, NON_VOTER 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) added. New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:19:01.162185 20354 consensus_peers.cc:485] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 -> Peer 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67:36255): Couldn't send request to peer 7c2550bd25424f3f82ad02367f52be2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f55c77486e5c42f99f05b43f82269720. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:01.174712 20287 consensus_queue.cc:237] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.176513 21319 raft_consensus.cc:2932] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, NON_VOTER f6f542b9bfba42028f61587b9d915542 (127.6.151.68) added. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } } }
I20240502 08:19:01.182041 20141 catalog_manager.cc:5484] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 reported cstate change: config changed from index -1 to 3, NON_VOTER f6f542b9bfba42028f61587b9d915542 (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 08:19:01.183692 20221 consensus_peers.cc:485] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 -> Peer f6f542b9bfba42028f61587b9d915542 (127.6.151.68:34235): Couldn't send request to peer f6f542b9bfba42028f61587b9d915542. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00a2296b66cd40f8ae82e8ca2354c065. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:01.476781 21375 ts_tablet_manager.cc:902] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: Initiating tablet copy from peer 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629)
I20240502 08:19:01.481808 21375 tablet_copy_client.cc:308] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:45629
I20240502 08:19:01.489225 20439 tablet_copy_service.cc:140] P 54eba73ef0bb466ab4776d6ea0541b78: Received BeginTabletCopySession request for tablet 8cacb00fa8b245a6b8604f90d8f5b618 from peer f6f542b9bfba42028f61587b9d915542 ({username='slave'} at 127.6.151.68:37561)
I20240502 08:19:01.489526 20439 tablet_copy_service.cc:161] P 54eba73ef0bb466ab4776d6ea0541b78: Beginning new tablet copy session on tablet 8cacb00fa8b245a6b8604f90d8f5b618 from peer f6f542b9bfba42028f61587b9d915542 at {username='slave'} at 127.6.151.68:37561: session id = f6f542b9bfba42028f61587b9d915542-8cacb00fa8b245a6b8604f90d8f5b618
I20240502 08:19:01.493268 20439 tablet_copy_source_session.cc:205] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:01.498402 21375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cacb00fa8b245a6b8604f90d8f5b618. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:01.510288 21375 tablet_copy_client.cc:783] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:01.510707 21375 tablet_copy_client.cc:647] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:01.513288 21375 tablet_copy_client.cc:523] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:01.517313 21375 tablet_bootstrap.cc:492] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: Bootstrap starting.
I20240502 08:19:01.524545 21375 log.cc:824] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:01.532732 21375 tablet_bootstrap.cc:492] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:01.533149 21375 tablet_bootstrap.cc:492] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: Bootstrap complete.
I20240502 08:19:01.533512 21375 ts_tablet_manager.cc:1367] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.005s
I20240502 08:19:01.537799 21375 raft_consensus.cc:348] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.538156 21375 raft_consensus.cc:729] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6f542b9bfba42028f61587b9d915542, State: Initialized, Role: LEARNER
I20240502 08:19:01.538565 21375 consensus_queue.cc:260] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.540083 21375 ts_tablet_manager.cc:1398] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20240502 08:19:01.541049 20439 tablet_copy_service.cc:342] P 54eba73ef0bb466ab4776d6ea0541b78: Request end of tablet copy session f6f542b9bfba42028f61587b9d915542-8cacb00fa8b245a6b8604f90d8f5b618 received from {username='slave'} at 127.6.151.68:37561
I20240502 08:19:01.541213 20439 tablet_copy_service.cc:433] P 54eba73ef0bb466ab4776d6ea0541b78: ending tablet copy session f6f542b9bfba42028f61587b9d915542-8cacb00fa8b245a6b8604f90d8f5b618 on tablet 8cacb00fa8b245a6b8604f90d8f5b618 with peer f6f542b9bfba42028f61587b9d915542
I20240502 08:19:01.590554 21400 ts_tablet_manager.cc:902] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: Initiating tablet copy from peer 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959)
I20240502 08:19:01.592633 21400 tablet_copy_client.cc:308] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:32959
I20240502 08:19:01.599395 20307 tablet_copy_service.cc:140] P 21f970fbfa08497f81784d2053e77171: Received BeginTabletCopySession request for tablet 711469afa64b4d4dbccaeb20a2e5f483 from peer 7c2550bd25424f3f82ad02367f52be2a ({username='slave'} at 127.6.151.67:46125)
I20240502 08:19:01.599668 20307 tablet_copy_service.cc:161] P 21f970fbfa08497f81784d2053e77171: Beginning new tablet copy session on tablet 711469afa64b4d4dbccaeb20a2e5f483 from peer 7c2550bd25424f3f82ad02367f52be2a at {username='slave'} at 127.6.151.67:46125: session id = 7c2550bd25424f3f82ad02367f52be2a-711469afa64b4d4dbccaeb20a2e5f483
I20240502 08:19:01.602335 20307 tablet_copy_source_session.cc:205] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:01.606575 21400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 711469afa64b4d4dbccaeb20a2e5f483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:01.613184 21406 ts_tablet_manager.cc:902] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: Initiating tablet copy from peer 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959)
I20240502 08:19:01.615154 21406 tablet_copy_client.cc:308] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:32959
I20240502 08:19:01.615967 20307 tablet_copy_service.cc:140] P 21f970fbfa08497f81784d2053e77171: Received BeginTabletCopySession request for tablet 7fd32afde3564645803d716a70853fe1 from peer 7c2550bd25424f3f82ad02367f52be2a ({username='slave'} at 127.6.151.67:46125)
I20240502 08:19:01.616158 20307 tablet_copy_service.cc:161] P 21f970fbfa08497f81784d2053e77171: Beginning new tablet copy session on tablet 7fd32afde3564645803d716a70853fe1 from peer 7c2550bd25424f3f82ad02367f52be2a at {username='slave'} at 127.6.151.67:46125: session id = 7c2550bd25424f3f82ad02367f52be2a-7fd32afde3564645803d716a70853fe1
I20240502 08:19:01.618274 20307 tablet_copy_source_session.cc:205] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:01.619220 21400 tablet_copy_client.cc:783] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:01.619586 21400 tablet_copy_client.cc:647] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:01.620116 21406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd32afde3564645803d716a70853fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:01.622257 21400 tablet_copy_client.cc:523] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:01.626856 21406 tablet_copy_client.cc:783] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:01.627195 21406 tablet_copy_client.cc:647] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:01.627173 21400 tablet_bootstrap.cc:492] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap starting.
I20240502 08:19:01.629056 21406 tablet_copy_client.cc:523] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:01.632573 21406 tablet_bootstrap.cc:492] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap starting.
I20240502 08:19:01.636175 21406 log.cc:824] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:01.636175 21400 log.cc:824] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:01.640638 21400 tablet_bootstrap.cc:492] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:01.640914 21406 tablet_bootstrap.cc:492] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:01.641325 21400 tablet_bootstrap.cc:492] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap complete.
I20240502 08:19:01.641466 21375 ts_tablet_manager.cc:902] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: Initiating tablet copy from peer 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629)
I20240502 08:19:01.641840 21400 ts_tablet_manager.cc:1367] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.003s
I20240502 08:19:01.642823 21406 tablet_bootstrap.cc:492] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap complete.
I20240502 08:19:01.643102 21406 ts_tablet_manager.cc:1367] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.001s
I20240502 08:19:01.644217 21375 tablet_copy_client.cc:308] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:45629
I20240502 08:19:01.644937 20439 tablet_copy_service.cc:140] P 54eba73ef0bb466ab4776d6ea0541b78: Received BeginTabletCopySession request for tablet 50581ab834944158a50bfc5a7c96b57c from peer f6f542b9bfba42028f61587b9d915542 ({username='slave'} at 127.6.151.68:37561)
I20240502 08:19:01.645110 20439 tablet_copy_service.cc:161] P 54eba73ef0bb466ab4776d6ea0541b78: Beginning new tablet copy session on tablet 50581ab834944158a50bfc5a7c96b57c from peer f6f542b9bfba42028f61587b9d915542 at {username='slave'} at 127.6.151.68:37561: session id = f6f542b9bfba42028f61587b9d915542-50581ab834944158a50bfc5a7c96b57c
I20240502 08:19:01.647038 20439 tablet_copy_source_session.cc:205] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:01.648247 21375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50581ab834944158a50bfc5a7c96b57c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:01.649796 21406 raft_consensus.cc:348] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.650233 21406 raft_consensus.cc:729] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c2550bd25424f3f82ad02367f52be2a, State: Initialized, Role: LEARNER
I20240502 08:19:01.650801 21406 consensus_queue.cc:260] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.652762 21406 ts_tablet_manager.cc:1398] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20240502 08:19:01.653506 21400 raft_consensus.cc:348] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.653743 21400 raft_consensus.cc:729] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c2550bd25424f3f82ad02367f52be2a, State: Initialized, Role: LEARNER
I20240502 08:19:01.653832 20307 tablet_copy_service.cc:342] P 21f970fbfa08497f81784d2053e77171: Request end of tablet copy session 7c2550bd25424f3f82ad02367f52be2a-7fd32afde3564645803d716a70853fe1 received from {username='slave'} at 127.6.151.67:46125
I20240502 08:19:01.654027 20307 tablet_copy_service.cc:433] P 21f970fbfa08497f81784d2053e77171: ending tablet copy session 7c2550bd25424f3f82ad02367f52be2a-7fd32afde3564645803d716a70853fe1 on tablet 7fd32afde3564645803d716a70853fe1 with peer 7c2550bd25424f3f82ad02367f52be2a
I20240502 08:19:01.653983 21400 consensus_queue.cc:260] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.654510 21400 ts_tablet_manager.cc:1398] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20240502 08:19:01.655565 20307 tablet_copy_service.cc:342] P 21f970fbfa08497f81784d2053e77171: Request end of tablet copy session 7c2550bd25424f3f82ad02367f52be2a-711469afa64b4d4dbccaeb20a2e5f483 received from {username='slave'} at 127.6.151.67:46125
I20240502 08:19:01.655709 20307 tablet_copy_service.cc:433] P 21f970fbfa08497f81784d2053e77171: ending tablet copy session 7c2550bd25424f3f82ad02367f52be2a-711469afa64b4d4dbccaeb20a2e5f483 on tablet 711469afa64b4d4dbccaeb20a2e5f483 with peer 7c2550bd25424f3f82ad02367f52be2a
I20240502 08:19:01.657740 21375 tablet_copy_client.cc:783] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:01.657785 21406 ts_tablet_manager.cc:902] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: Initiating tablet copy from peer 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629)
I20240502 08:19:01.657991 21375 tablet_copy_client.cc:647] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:01.659195 21406 tablet_copy_client.cc:308] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:45629
I20240502 08:19:01.659698 21375 tablet_copy_client.cc:523] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:01.663168 21375 tablet_bootstrap.cc:492] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: Bootstrap starting.
I20240502 08:19:01.665755 20439 tablet_copy_service.cc:140] P 54eba73ef0bb466ab4776d6ea0541b78: Received BeginTabletCopySession request for tablet f55c77486e5c42f99f05b43f82269720 from peer 7c2550bd25424f3f82ad02367f52be2a ({username='slave'} at 127.6.151.67:43455)
I20240502 08:19:01.665908 20439 tablet_copy_service.cc:161] P 54eba73ef0bb466ab4776d6ea0541b78: Beginning new tablet copy session on tablet f55c77486e5c42f99f05b43f82269720 from peer 7c2550bd25424f3f82ad02367f52be2a at {username='slave'} at 127.6.151.67:43455: session id = 7c2550bd25424f3f82ad02367f52be2a-f55c77486e5c42f99f05b43f82269720
I20240502 08:19:01.668258 20439 tablet_copy_source_session.cc:205] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:01.668655 21375 tablet_bootstrap.cc:492] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:01.669041 21375 tablet_bootstrap.cc:492] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: Bootstrap complete.
I20240502 08:19:01.669315 21375 ts_tablet_manager.cc:1367] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 08:19:01.669574 21406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55c77486e5c42f99f05b43f82269720. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:01.670338 21375 raft_consensus.cc:348] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.670598 21375 raft_consensus.cc:729] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6f542b9bfba42028f61587b9d915542, State: Initialized, Role: LEARNER
I20240502 08:19:01.670825 21375 consensus_queue.cc:260] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.671409 21375 ts_tablet_manager.cc:1398] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542: Time spent starting tablet: real 0.002s	user 0.003s	sys 0.000s
I20240502 08:19:01.672204 20439 tablet_copy_service.cc:342] P 54eba73ef0bb466ab4776d6ea0541b78: Request end of tablet copy session f6f542b9bfba42028f61587b9d915542-50581ab834944158a50bfc5a7c96b57c received from {username='slave'} at 127.6.151.68:37561
I20240502 08:19:01.672374 20439 tablet_copy_service.cc:433] P 54eba73ef0bb466ab4776d6ea0541b78: ending tablet copy session f6f542b9bfba42028f61587b9d915542-50581ab834944158a50bfc5a7c96b57c on tablet 50581ab834944158a50bfc5a7c96b57c with peer f6f542b9bfba42028f61587b9d915542
I20240502 08:19:01.674973 21406 tablet_copy_client.cc:783] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:01.675189 21406 tablet_copy_client.cc:647] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:01.676693 21406 tablet_copy_client.cc:523] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:01.678974 21406 tablet_bootstrap.cc:492] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap starting.
I20240502 08:19:01.683436 21406 tablet_bootstrap.cc:492] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:01.683753 21406 tablet_bootstrap.cc:492] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: Bootstrap complete.
I20240502 08:19:01.683966 21406 ts_tablet_manager.cc:1367] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.001s
I20240502 08:19:01.684547 21406 raft_consensus.cc:348] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.684768 21406 raft_consensus.cc:729] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7c2550bd25424f3f82ad02367f52be2a, State: Initialized, Role: LEARNER
I20240502 08:19:01.684975 21406 consensus_queue.cc:260] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: NON_VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: true } }
I20240502 08:19:01.685487 21406 ts_tablet_manager.cc:1398] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:01.686300 20439 tablet_copy_service.cc:342] P 54eba73ef0bb466ab4776d6ea0541b78: Request end of tablet copy session 7c2550bd25424f3f82ad02367f52be2a-f55c77486e5c42f99f05b43f82269720 received from {username='slave'} at 127.6.151.67:43455
I20240502 08:19:01.686424 20439 tablet_copy_service.cc:433] P 54eba73ef0bb466ab4776d6ea0541b78: ending tablet copy session 7c2550bd25424f3f82ad02367f52be2a-f55c77486e5c42f99f05b43f82269720 on tablet f55c77486e5c42f99f05b43f82269720 with peer 7c2550bd25424f3f82ad02367f52be2a
I20240502 08:19:01.760193 21375 ts_tablet_manager.cc:902] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: Initiating tablet copy from peer 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959)
I20240502 08:19:01.769841 21375 tablet_copy_client.cc:308] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:32959
I20240502 08:19:01.776736 20307 tablet_copy_service.cc:140] P 21f970fbfa08497f81784d2053e77171: Received BeginTabletCopySession request for tablet 00a2296b66cd40f8ae82e8ca2354c065 from peer f6f542b9bfba42028f61587b9d915542 ({username='slave'} at 127.6.151.68:56475)
I20240502 08:19:01.776935 20307 tablet_copy_service.cc:161] P 21f970fbfa08497f81784d2053e77171: Beginning new tablet copy session on tablet 00a2296b66cd40f8ae82e8ca2354c065 from peer f6f542b9bfba42028f61587b9d915542 at {username='slave'} at 127.6.151.68:56475: session id = f6f542b9bfba42028f61587b9d915542-00a2296b66cd40f8ae82e8ca2354c065
I20240502 08:19:01.779162 20307 tablet_copy_source_session.cc:205] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:01.780401 21375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00a2296b66cd40f8ae82e8ca2354c065. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:01.785650 21375 tablet_copy_client.cc:783] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:01.785866 21375 tablet_copy_client.cc:647] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:01.787312 21375 tablet_copy_client.cc:523] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:01.789604 21375 tablet_bootstrap.cc:492] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: Bootstrap starting.
I20240502 08:19:01.795178 21375 tablet_bootstrap.cc:492] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:01.795532 21375 tablet_bootstrap.cc:492] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: Bootstrap complete.
I20240502 08:19:01.795753 21375 ts_tablet_manager.cc:1367] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: Time spent bootstrapping tablet: real 0.006s	user 0.007s	sys 0.000s
I20240502 08:19:01.796324 21375 raft_consensus.cc:348] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.796506 21375 raft_consensus.cc:729] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f6f542b9bfba42028f61587b9d915542, State: Initialized, Role: LEARNER
I20240502 08:19:01.796713 21375 consensus_queue.cc:260] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: true } }
I20240502 08:19:01.797235 21375 ts_tablet_manager.cc:1398] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:01.798846 20307 tablet_copy_service.cc:342] P 21f970fbfa08497f81784d2053e77171: Request end of tablet copy session f6f542b9bfba42028f61587b9d915542-00a2296b66cd40f8ae82e8ca2354c065 received from {username='slave'} at 127.6.151.68:56475
I20240502 08:19:01.799012 20307 tablet_copy_service.cc:433] P 21f970fbfa08497f81784d2053e77171: ending tablet copy session f6f542b9bfba42028f61587b9d915542-00a2296b66cd40f8ae82e8ca2354c065 on tablet 00a2296b66cd40f8ae82e8ca2354c065 with peer f6f542b9bfba42028f61587b9d915542
I20240502 08:19:02.046324 20994 raft_consensus.cc:1203] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240502 08:19:02.050578 21126 raft_consensus.cc:1203] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240502 08:19:02.087668 20994 raft_consensus.cc:1203] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240502 08:19:02.104863 20994 raft_consensus.cc:1203] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240502 08:19:02.146448 21126 raft_consensus.cc:1203] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3]   Dedup: 1.3->[]
I20240502 08:19:02.154253 21126 raft_consensus.cc:1203] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20240502 08:19:02.486958 21492 raft_consensus.cc:1050] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: attempting to promote NON_VOTER 7c2550bd25424f3f82ad02367f52be2a to VOTER
I20240502 08:19:02.487779 21492 consensus_queue.cc:237] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:02.489799 20994 raft_consensus.cc:1261] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Refusing update from remote peer 21f970fbfa08497f81784d2053e77171: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:02.490458 21491 consensus_queue.cc:1035] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:02.494129 21492 raft_consensus.cc:2932] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:02.494868 20994 raft_consensus.cc:2932] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:02.498363 20141 catalog_manager.cc:5484] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 reported cstate change: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:02.506008 21494 raft_consensus.cc:1050] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: attempting to promote NON_VOTER f6f542b9bfba42028f61587b9d915542 to VOTER
I20240502 08:19:02.506866 21494 consensus_queue.cc:237] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:02.508937 21126 raft_consensus.cc:1261] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Refusing update from remote peer 54eba73ef0bb466ab4776d6ea0541b78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:02.509677 21493 consensus_queue.cc:1035] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:02.513576 21494 raft_consensus.cc:2932] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:02.514286 21126 raft_consensus.cc:2932] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:02.518204 20145 catalog_manager.cc:5484] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: config changed from index 3 to 4, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:02.559888 20419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8cacb00fa8b245a6b8604f90d8f5b618"
dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34588
I20240502 08:19:02.560158 20419 raft_consensus.cc:595] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:02.564324 21311 raft_consensus.cc:1050] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: attempting to promote NON_VOTER 7c2550bd25424f3f82ad02367f52be2a to VOTER
I20240502 08:19:02.564932 21311 consensus_queue.cc:237] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:02.566859 20994 raft_consensus.cc:1261] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Refusing update from remote peer 54eba73ef0bb466ab4776d6ea0541b78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:02.567430 21311 consensus_queue.cc:1035] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:02.570125 21494 raft_consensus.cc:2932] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:02.570946 20994 raft_consensus.cc:2932] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:02.575598 20141 catalog_manager.cc:5484] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:02.582878 20419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f55c77486e5c42f99f05b43f82269720"
dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34608
I20240502 08:19:02.583082 20419 raft_consensus.cc:595] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:02.594841 21491 raft_consensus.cc:1050] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: attempting to promote NON_VOTER 7c2550bd25424f3f82ad02367f52be2a to VOTER
I20240502 08:19:02.595582 21491 consensus_queue.cc:237] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:02.598168 20994 raft_consensus.cc:1261] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 1 LEARNER]: Refusing update from remote peer 21f970fbfa08497f81784d2053e77171: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:02.598731 21319 consensus_queue.cc:1035] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:02.600644 21492 raft_consensus.cc:1050] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: attempting to promote NON_VOTER f6f542b9bfba42028f61587b9d915542 to VOTER
I20240502 08:19:02.601296 21492 consensus_queue.cc:237] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:02.601895 21582 raft_consensus.cc:2932] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:02.603255 20994 raft_consensus.cc:2932] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:02.603984 21126 raft_consensus.cc:1261] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Refusing update from remote peer 21f970fbfa08497f81784d2053e77171: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:02.604516 21492 consensus_queue.cc:1035] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:02.604658 20287 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7fd32afde3564645803d716a70853fe1"
dest_uuid: "21f970fbfa08497f81784d2053e77171"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59496
I20240502 08:19:02.604935 20287 raft_consensus.cc:595] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:02.608793 20145 catalog_manager.cc:5484] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a reported cstate change: config changed from index 2 to 3, 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:02.612067 21126 raft_consensus.cc:2932] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:02.611364 21491 raft_consensus.cc:2932] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:02.619760 20141 catalog_manager.cc:5484] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 reported cstate change: config changed from index 3 to 4, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "21f970fbfa08497f81784d2053e77171" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:02.770022 21493 raft_consensus.cc:1050] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: attempting to promote NON_VOTER f6f542b9bfba42028f61587b9d915542 to VOTER
I20240502 08:19:02.770802 21493 consensus_queue.cc:237] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:02.772810 21126 raft_consensus.cc:1261] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 1 LEARNER]: Refusing update from remote peer 54eba73ef0bb466ab4776d6ea0541b78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 08:19:02.773363 21311 consensus_queue.cc:1035] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 08:19:02.776098 21493 raft_consensus.cc:2932] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:02.776886 21126 raft_consensus.cc:2932] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:02.781770 20145 catalog_manager.cc:5484] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 reported cstate change: config changed from index 2 to 3, f6f542b9bfba42028f61587b9d915542 (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "54eba73ef0bb466ab4776d6ea0541b78" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:02.828357 20287 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "00a2296b66cd40f8ae82e8ca2354c065"
dest_uuid: "21f970fbfa08497f81784d2053e77171"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59526
I20240502 08:19:02.828644 20287 raft_consensus.cc:595] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:02.850510 20419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50581ab834944158a50bfc5a7c96b57c"
dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34632
I20240502 08:19:02.850760 20419 raft_consensus.cc:595] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:02.870609 20287 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "711469afa64b4d4dbccaeb20a2e5f483"
dest_uuid: "21f970fbfa08497f81784d2053e77171"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59546
I20240502 08:19:02.870993 20287 raft_consensus.cc:595] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:02.886322 21582 raft_consensus.cc:979] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: : Instructing follower 7c2550bd25424f3f82ad02367f52be2a to start an election
I20240502 08:19:02.886489 21582 raft_consensus.cc:1067] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Signalling peer 7c2550bd25424f3f82ad02367f52be2a to start an election
I20240502 08:19:02.887374 20994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7fd32afde3564645803d716a70853fe1"
dest_uuid: "7c2550bd25424f3f82ad02367f52be2a"
 from {username='slave'} at 127.6.151.65:42823
I20240502 08:19:02.887712 20994 raft_consensus.cc:482] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:02.887920 20994 raft_consensus.cc:3037] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:02.890374 20994 raft_consensus.cc:504] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:02.891209 20419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8cacb00fa8b245a6b8604f90d8f5b618"
dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34644
I20240502 08:19:02.891412 20419 raft_consensus.cc:595] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:02.891834 20994 leader_election.cc:288] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [CANDIDATE]: Term 2 election: Requested vote from peers 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959)
I20240502 08:19:02.892608 20287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd32afde3564645803d716a70853fe1" candidate_uuid: "7c2550bd25424f3f82ad02367f52be2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "21f970fbfa08497f81784d2053e77171"
I20240502 08:19:02.892971 20287 raft_consensus.cc:3032] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:02.893060 20287 raft_consensus.cc:729] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:19:02.893410 20287 consensus_queue.cc:260] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:02.893790 20287 raft_consensus.cc:3037] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:02.896385 20287 raft_consensus.cc:2454] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2550bd25424f3f82ad02367f52be2a in term 2.
I20240502 08:19:02.896961 20929 leader_election.cc:302] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171, 7c2550bd25424f3f82ad02367f52be2a; no voters: 
I20240502 08:19:02.897389 21418 raft_consensus.cc:2781] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:02.897861 21418 raft_consensus.cc:686] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 2 LEADER]: Becoming Leader. State: Replica: 7c2550bd25424f3f82ad02367f52be2a, State: Running, Role: LEADER
I20240502 08:19:02.898339 21418 consensus_queue.cc:237] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:02.901968 20141 catalog_manager.cc:5484] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a reported cstate change: term changed from 1 to 2, leader changed from 21f970fbfa08497f81784d2053e77171 (127.6.151.65) to 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67). New cstate: current_term: 2 leader_uuid: "7c2550bd25424f3f82ad02367f52be2a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 08:19:02.913414 20131 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 7fd32afde3564645803d716a70853fe1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67:36255) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 08:19:02.915270 20419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f55c77486e5c42f99f05b43f82269720"
dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34662
I20240502 08:19:02.915498 20419 raft_consensus.cc:595] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:03.064052 21494 raft_consensus.cc:979] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: : Instructing follower 7c2550bd25424f3f82ad02367f52be2a to start an election
I20240502 08:19:03.064271 21494 raft_consensus.cc:1067] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Signalling peer 7c2550bd25424f3f82ad02367f52be2a to start an election
I20240502 08:19:03.065125 20994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f55c77486e5c42f99f05b43f82269720"
dest_uuid: "7c2550bd25424f3f82ad02367f52be2a"
 from {username='slave'} at 127.6.151.66:34637
I20240502 08:19:03.065361 20994 raft_consensus.cc:482] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:03.065490 20994 raft_consensus.cc:3037] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.067451 20994 raft_consensus.cc:504] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.068065 20994 leader_election.cc:288] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [CANDIDATE]: Term 2 election: Requested vote from peers 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629)
I20240502 08:19:03.068749 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55c77486e5c42f99f05b43f82269720" candidate_uuid: "7c2550bd25424f3f82ad02367f52be2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
I20240502 08:19:03.069032 20419 raft_consensus.cc:3032] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:03.069115 20419 raft_consensus.cc:729] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:19:03.069481 20419 consensus_queue.cc:260] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.069864 20419 raft_consensus.cc:3037] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.072008 20419 raft_consensus.cc:2454] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2550bd25424f3f82ad02367f52be2a in term 2.
I20240502 08:19:03.072510 20930 leader_election.cc:302] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78, 7c2550bd25424f3f82ad02367f52be2a; no voters: 
I20240502 08:19:03.072842 21418 raft_consensus.cc:2781] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:03.073001 21418 raft_consensus.cc:686] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 2 LEADER]: Becoming Leader. State: Replica: 7c2550bd25424f3f82ad02367f52be2a, State: Running, Role: LEADER
I20240502 08:19:03.073282 21418 consensus_queue.cc:237] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.075489 20141 catalog_manager.cc:5484] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a reported cstate change: term changed from 1 to 2, leader changed from 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) to 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67). New cstate: current_term: 2 leader_uuid: "7c2550bd25424f3f82ad02367f52be2a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.112447 21491 raft_consensus.cc:979] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: : Instructing follower f6f542b9bfba42028f61587b9d915542 to start an election
I20240502 08:19:03.112628 21491 raft_consensus.cc:1067] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Signalling peer f6f542b9bfba42028f61587b9d915542 to start an election
I20240502 08:19:03.113493 21126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00a2296b66cd40f8ae82e8ca2354c065"
dest_uuid: "f6f542b9bfba42028f61587b9d915542"
 from {username='slave'} at 127.6.151.65:52159
I20240502 08:19:03.113797 21126 raft_consensus.cc:482] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:03.113953 21126 raft_consensus.cc:3037] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.116179 21126 raft_consensus.cc:504] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.117513 21126 leader_election.cc:288] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [CANDIDATE]: Term 2 election: Requested vote from peers 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959)
I20240502 08:19:03.118113 20287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00a2296b66cd40f8ae82e8ca2354c065" candidate_uuid: "f6f542b9bfba42028f61587b9d915542" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "21f970fbfa08497f81784d2053e77171"
I20240502 08:19:03.118306 20287 raft_consensus.cc:3032] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:03.118372 20287 raft_consensus.cc:729] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:19:03.118618 20287 consensus_queue.cc:260] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.118942 20287 raft_consensus.cc:3037] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.120823 20287 raft_consensus.cc:2454] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f542b9bfba42028f61587b9d915542 in term 2.
I20240502 08:19:03.121348 21060 leader_election.cc:302] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171, f6f542b9bfba42028f61587b9d915542; no voters: 
I20240502 08:19:03.121699 21399 raft_consensus.cc:2781] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:03.122174 21399 raft_consensus.cc:686] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 2 LEADER]: Becoming Leader. State: Replica: f6f542b9bfba42028f61587b9d915542, State: Running, Role: LEADER
I20240502 08:19:03.122674 21399 consensus_queue.cc:237] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.126178 20141 catalog_manager.cc:5484] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 reported cstate change: term changed from 1 to 2, leader changed from 21f970fbfa08497f81784d2053e77171 (127.6.151.65) to f6f542b9bfba42028f61587b9d915542 (127.6.151.68). New cstate: current_term: 2 leader_uuid: "f6f542b9bfba42028f61587b9d915542" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.131950 21494 raft_consensus.cc:979] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: : Instructing follower f6f542b9bfba42028f61587b9d915542 to start an election
I20240502 08:19:03.132129 21494 raft_consensus.cc:1067] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Signalling peer f6f542b9bfba42028f61587b9d915542 to start an election
I20240502 08:19:03.132990 21126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cacb00fa8b245a6b8604f90d8f5b618"
dest_uuid: "f6f542b9bfba42028f61587b9d915542"
 from {username='slave'} at 127.6.151.66:50897
I20240502 08:19:03.133298 21126 raft_consensus.cc:482] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:03.133472 21126 raft_consensus.cc:3037] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.140475 21126 raft_consensus.cc:504] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.141278 21126 leader_election.cc:288] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [CANDIDATE]: Term 2 election: Requested vote from peers 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629)
I20240502 08:19:03.141894 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cacb00fa8b245a6b8604f90d8f5b618" candidate_uuid: "f6f542b9bfba42028f61587b9d915542" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
I20240502 08:19:03.142128 20419 raft_consensus.cc:3032] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:03.142216 20419 raft_consensus.cc:729] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:19:03.142529 20419 consensus_queue.cc:260] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.142881 20419 raft_consensus.cc:3037] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.145453 20419 raft_consensus.cc:2454] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f542b9bfba42028f61587b9d915542 in term 2.
I20240502 08:19:03.145893 21061 leader_election.cc:302] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78, f6f542b9bfba42028f61587b9d915542; no voters: 
I20240502 08:19:03.146167 21399 raft_consensus.cc:2781] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:03.146342 21399 raft_consensus.cc:686] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 2 LEADER]: Becoming Leader. State: Replica: f6f542b9bfba42028f61587b9d915542, State: Running, Role: LEADER
I20240502 08:19:03.146606 21399 consensus_queue.cc:237] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.148975 20145 catalog_manager.cc:5484] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 reported cstate change: term changed from 1 to 2, leader changed from 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) to f6f542b9bfba42028f61587b9d915542 (127.6.151.68). New cstate: current_term: 2 leader_uuid: "f6f542b9bfba42028f61587b9d915542" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.157150 21582 raft_consensus.cc:979] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: : Instructing follower 7c2550bd25424f3f82ad02367f52be2a to start an election
I20240502 08:19:03.157368 21582 raft_consensus.cc:1067] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Signalling peer 7c2550bd25424f3f82ad02367f52be2a to start an election
I20240502 08:19:03.158087 20994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "711469afa64b4d4dbccaeb20a2e5f483"
dest_uuid: "7c2550bd25424f3f82ad02367f52be2a"
 from {username='slave'} at 127.6.151.65:42823
I20240502 08:19:03.158337 20994 raft_consensus.cc:482] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:03.158463 20994 raft_consensus.cc:3037] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.160626 20994 raft_consensus.cc:504] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.161574 20994 leader_election.cc:288] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [CANDIDATE]: Term 2 election: Requested vote from peers 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959)
I20240502 08:19:03.162137 20287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "711469afa64b4d4dbccaeb20a2e5f483" candidate_uuid: "7c2550bd25424f3f82ad02367f52be2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "21f970fbfa08497f81784d2053e77171"
I20240502 08:19:03.162355 20287 raft_consensus.cc:3032] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:03.162444 20287 raft_consensus.cc:729] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 21f970fbfa08497f81784d2053e77171, State: Running, Role: LEADER
I20240502 08:19:03.162786 20287 consensus_queue.cc:260] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.163074 20287 raft_consensus.cc:3037] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.164608 21493 raft_consensus.cc:979] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: : Instructing follower f6f542b9bfba42028f61587b9d915542 to start an election
I20240502 08:19:03.164793 21311 raft_consensus.cc:1067] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Signalling peer f6f542b9bfba42028f61587b9d915542 to start an election
I20240502 08:19:03.165426 21126 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50581ab834944158a50bfc5a7c96b57c"
dest_uuid: "f6f542b9bfba42028f61587b9d915542"
 from {username='slave'} at 127.6.151.66:50897
I20240502 08:19:03.165637 21126 raft_consensus.cc:482] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:03.165745 21126 raft_consensus.cc:3037] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.165827 20287 raft_consensus.cc:2454] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c2550bd25424f3f82ad02367f52be2a in term 2.
I20240502 08:19:03.166391 20929 leader_election.cc:302] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 21f970fbfa08497f81784d2053e77171, 7c2550bd25424f3f82ad02367f52be2a; no voters: 
I20240502 08:19:03.166877 21418 raft_consensus.cc:2781] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:03.167063 21418 raft_consensus.cc:686] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 2 LEADER]: Becoming Leader. State: Replica: 7c2550bd25424f3f82ad02367f52be2a, State: Running, Role: LEADER
I20240502 08:19:03.167376 21418 consensus_queue.cc:237] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.168645 20419 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "50581ab834944158a50bfc5a7c96b57c"
dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:34684
I20240502 08:19:03.168893 20419 raft_consensus.cc:595] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:03.169318 21126 raft_consensus.cc:504] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.170301 20419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50581ab834944158a50bfc5a7c96b57c" candidate_uuid: "f6f542b9bfba42028f61587b9d915542" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "54eba73ef0bb466ab4776d6ea0541b78"
I20240502 08:19:03.170215 20145 catalog_manager.cc:5484] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a reported cstate change: term changed from 1 to 2, leader changed from 21f970fbfa08497f81784d2053e77171 (127.6.151.65) to 7c2550bd25424f3f82ad02367f52be2a (127.6.151.67). New cstate: current_term: 2 leader_uuid: "7c2550bd25424f3f82ad02367f52be2a" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.170533 20419 raft_consensus.cc:3032] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:03.170609 20419 raft_consensus.cc:729] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 54eba73ef0bb466ab4776d6ea0541b78, State: Running, Role: LEADER
I20240502 08:19:03.170902 20419 consensus_queue.cc:260] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.171133 20419 raft_consensus.cc:3037] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:03.174146 20419 raft_consensus.cc:2454] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f542b9bfba42028f61587b9d915542 in term 2.
I20240502 08:19:03.174602 21061 leader_election.cc:302] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 54eba73ef0bb466ab4776d6ea0541b78, f6f542b9bfba42028f61587b9d915542; no voters: 
I20240502 08:19:03.174975 21399 raft_consensus.cc:2781] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:03.175096 21399 raft_consensus.cc:686] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 2 LEADER]: Becoming Leader. State: Replica: f6f542b9bfba42028f61587b9d915542, State: Running, Role: LEADER
I20240502 08:19:03.175381 21399 consensus_queue.cc:237] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.176209 21126 leader_election.cc:288] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [CANDIDATE]: Term 2 election: Requested vote from peers 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629)
I20240502 08:19:03.178105 20145 catalog_manager.cc:5484] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 reported cstate change: term changed from 1 to 2, leader changed from 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) to f6f542b9bfba42028f61587b9d915542 (127.6.151.68). New cstate: current_term: 2 leader_uuid: "f6f542b9bfba42028f61587b9d915542" committed_config { opid_index: 3 OBSOLETE_local: true peers { permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.320935 20287 raft_consensus.cc:1261] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Refusing update from remote peer 7c2550bd25424f3f82ad02367f52be2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:03.321861 21418 consensus_queue.cc:1035] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:03.555467 20994 consensus_queue.cc:237] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.556429 21627 raft_consensus.cc:2932] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 21f970fbfa08497f81784d2053e77171 (127.6.151.65) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:03.559231 20131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7fd32afde3564645803d716a70853fe1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:19:03.560683 20141 catalog_manager.cc:5484] T 7fd32afde3564645803d716a70853fe1 P 7c2550bd25424f3f82ad02367f52be2a reported cstate change: config changed from index 3 to 5, VOTER 21f970fbfa08497f81784d2053e77171 (127.6.151.65) evicted. New cstate: current_term: 2 leader_uuid: "7c2550bd25424f3f82ad02367f52be2a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.564585 20266 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd32afde3564645803d716a70853fe1 with delete_type TABLET_DATA_TOMBSTONED (TS 21f970fbfa08497f81784d2053e77171 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39860
I20240502 08:19:03.565469 21716 tablet_replica.cc:330] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: stopping tablet replica
I20240502 08:19:03.565850 21716 raft_consensus.cc:2229] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:03.566097 21716 raft_consensus.cc:2258] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:03.567448 21716 ts_tablet_manager.cc:1875] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:03.572157 21716 ts_tablet_manager.cc:1888] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:19:03.572310 21716 log.cc:1196] T 7fd32afde3564645803d716a70853fe1 P 21f970fbfa08497f81784d2053e77171: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/7fd32afde3564645803d716a70853fe1
I20240502 08:19:03.573184 20131 catalog_manager.cc:4830] TS 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959): tablet 7fd32afde3564645803d716a70853fe1 (table rebalance_test_table_1 [id=45d6e535683449bfb87fccecd5fc1061]) successfully deleted
I20240502 08:19:03.613955 20419 raft_consensus.cc:1261] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Refusing update from remote peer 7c2550bd25424f3f82ad02367f52be2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:03.614737 21627 consensus_queue.cc:1035] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:03.642968 20419 raft_consensus.cc:1261] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Refusing update from remote peer f6f542b9bfba42028f61587b9d915542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:19:03.643620 21399 consensus_queue.cc:1035] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:03.659070 20287 raft_consensus.cc:1261] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Refusing update from remote peer 7c2550bd25424f3f82ad02367f52be2a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:03.659653 21627 consensus_queue.cc:1035] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:03.696223 20287 raft_consensus.cc:1261] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Refusing update from remote peer f6f542b9bfba42028f61587b9d915542: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:19:03.696952 21645 consensus_queue.cc:1035] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f970fbfa08497f81784d2053e77171" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 32959 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:03.697497 20419 raft_consensus.cc:1261] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Refusing update from remote peer f6f542b9bfba42028f61587b9d915542: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 08:19:03.698700 21645 consensus_queue.cc:1035] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54eba73ef0bb466ab4776d6ea0541b78" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 45629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:03.713179 20994 consensus_queue.cc:237] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.714121 21418 raft_consensus.cc:2932] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:03.718185 20131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f55c77486e5c42f99f05b43f82269720 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:19:03.719528 20145 catalog_manager.cc:5484] T f55c77486e5c42f99f05b43f82269720 P 7c2550bd25424f3f82ad02367f52be2a reported cstate change: config changed from index 3 to 5, VOTER 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "7c2550bd25424f3f82ad02367f52be2a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.723426 20399 tablet_service.cc:1514] Processing DeleteTablet for tablet f55c77486e5c42f99f05b43f82269720 with delete_type TABLET_DATA_TOMBSTONED (TS 54eba73ef0bb466ab4776d6ea0541b78 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34548
I20240502 08:19:03.724336 21745 tablet_replica.cc:330] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: stopping tablet replica
I20240502 08:19:03.724886 21745 raft_consensus.cc:2229] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:03.725279 21745 raft_consensus.cc:2258] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:03.726936 21745 ts_tablet_manager.cc:1875] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:03.733232 21745 ts_tablet_manager.cc:1888] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:19:03.733376 21745 log.cc:1196] T f55c77486e5c42f99f05b43f82269720 P 54eba73ef0bb466ab4776d6ea0541b78: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/f55c77486e5c42f99f05b43f82269720
I20240502 08:19:03.734982 20132 catalog_manager.cc:4830] TS 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629): tablet f55c77486e5c42f99f05b43f82269720 (table rebalance_test_table_0 [id=1dd7c764b67a4fe7bfba877c635c3701]) successfully deleted
I20240502 08:19:03.777230 21126 consensus_queue.cc:237] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.778177 21734 raft_consensus.cc:2932] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:03.781404 20131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50581ab834944158a50bfc5a7c96b57c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:19:03.782866 20145 catalog_manager.cc:5484] T 50581ab834944158a50bfc5a7c96b57c P f6f542b9bfba42028f61587b9d915542 reported cstate change: config changed from index 3 to 5, VOTER 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "f6f542b9bfba42028f61587b9d915542" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.786832 20399 tablet_service.cc:1514] Processing DeleteTablet for tablet 50581ab834944158a50bfc5a7c96b57c with delete_type TABLET_DATA_TOMBSTONED (TS 54eba73ef0bb466ab4776d6ea0541b78 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34548
I20240502 08:19:03.787251 21745 tablet_replica.cc:330] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: stopping tablet replica
I20240502 08:19:03.787443 21745 raft_consensus.cc:2229] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:03.787639 21745 raft_consensus.cc:2258] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:03.788528 21745 ts_tablet_manager.cc:1875] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:03.794173 21125 consensus_queue.cc:237] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.794358 21745 ts_tablet_manager.cc:1888] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:19:03.794481 21745 log.cc:1196] T 50581ab834944158a50bfc5a7c96b57c P 54eba73ef0bb466ab4776d6ea0541b78: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/50581ab834944158a50bfc5a7c96b57c
I20240502 08:19:03.795379 20132 catalog_manager.cc:4830] TS 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629): tablet 50581ab834944158a50bfc5a7c96b57c (table rebalance_test_table_1 [id=45d6e535683449bfb87fccecd5fc1061]) successfully deleted
I20240502 08:19:03.795258 21734 raft_consensus.cc:2932] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 21f970fbfa08497f81784d2053e77171 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:03.797919 20131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00a2296b66cd40f8ae82e8ca2354c065 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:19:03.799291 20145 catalog_manager.cc:5484] T 00a2296b66cd40f8ae82e8ca2354c065 P f6f542b9bfba42028f61587b9d915542 reported cstate change: config changed from index 4 to 6, VOTER 21f970fbfa08497f81784d2053e77171 (127.6.151.65) evicted. New cstate: current_term: 2 leader_uuid: "f6f542b9bfba42028f61587b9d915542" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.802765 20266 tablet_service.cc:1514] Processing DeleteTablet for tablet 00a2296b66cd40f8ae82e8ca2354c065 with delete_type TABLET_DATA_TOMBSTONED (TS 21f970fbfa08497f81784d2053e77171 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39860
I20240502 08:19:03.803130 21716 tablet_replica.cc:330] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: stopping tablet replica
I20240502 08:19:03.803318 21716 raft_consensus.cc:2229] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:03.803508 21716 raft_consensus.cc:2258] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:03.804250 21716 ts_tablet_manager.cc:1875] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:03.808478 21716 ts_tablet_manager.cc:1888] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:19:03.808609 21716 log.cc:1196] T 00a2296b66cd40f8ae82e8ca2354c065 P 21f970fbfa08497f81784d2053e77171: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/00a2296b66cd40f8ae82e8ca2354c065
I20240502 08:19:03.809953 20131 catalog_manager.cc:4830] TS 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959): tablet 00a2296b66cd40f8ae82e8ca2354c065 (table rebalance_test_table_3 [id=8928a9601899448ca215c57c7857f8e6]) successfully deleted
I20240502 08:19:03.819790 20994 consensus_queue.cc:237] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } }
I20240502 08:19:03.820607 21418 raft_consensus.cc:2932] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 21f970fbfa08497f81784d2053e77171 (127.6.151.65) evicted. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } } }
I20240502 08:19:03.823408 20131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 711469afa64b4d4dbccaeb20a2e5f483 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:19:03.824834 20145 catalog_manager.cc:5484] T 711469afa64b4d4dbccaeb20a2e5f483 P 7c2550bd25424f3f82ad02367f52be2a reported cstate change: config changed from index 3 to 5, VOTER 21f970fbfa08497f81784d2053e77171 (127.6.151.65) evicted. New cstate: current_term: 2 leader_uuid: "7c2550bd25424f3f82ad02367f52be2a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7c2550bd25424f3f82ad02367f52be2a" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.828146 21125 consensus_queue.cc:237] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } }
I20240502 08:19:03.828397 20266 tablet_service.cc:1514] Processing DeleteTablet for tablet 711469afa64b4d4dbccaeb20a2e5f483 with delete_type TABLET_DATA_TOMBSTONED (TS 21f970fbfa08497f81784d2053e77171 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39860
I20240502 08:19:03.828771 21716 tablet_replica.cc:330] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: stopping tablet replica
I20240502 08:19:03.828977 21716 raft_consensus.cc:2229] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:03.829193 21716 raft_consensus.cc:2258] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:03.828989 21734 raft_consensus.cc:2932] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } } }
I20240502 08:19:03.830024 21716 ts_tablet_manager.cc:1875] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:03.832968 20131 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cacb00fa8b245a6b8604f90d8f5b618 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 08:19:03.833750 20145 catalog_manager.cc:5484] T 8cacb00fa8b245a6b8604f90d8f5b618 P f6f542b9bfba42028f61587b9d915542 reported cstate change: config changed from index 4 to 6, VOTER 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66) evicted. New cstate: current_term: 2 leader_uuid: "f6f542b9bfba42028f61587b9d915542" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f6f542b9bfba42028f61587b9d915542" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34235 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:03.836882 21716 ts_tablet_manager.cc:1888] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 08:19:03.837066 21716 log.cc:1196] T 711469afa64b4d4dbccaeb20a2e5f483 P 21f970fbfa08497f81784d2053e77171: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/711469afa64b4d4dbccaeb20a2e5f483
I20240502 08:19:03.837090 20399 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cacb00fa8b245a6b8604f90d8f5b618 with delete_type TABLET_DATA_TOMBSTONED (TS 54eba73ef0bb466ab4776d6ea0541b78 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34548
I20240502 08:19:03.837507 21745 tablet_replica.cc:330] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: stopping tablet replica
I20240502 08:19:03.837731 21745 raft_consensus.cc:2229] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:03.837855 20131 catalog_manager.cc:4830] TS 21f970fbfa08497f81784d2053e77171 (127.6.151.65:32959): tablet 711469afa64b4d4dbccaeb20a2e5f483 (table TestTable [id=8db3be422c3b4045b1568952325b4fe0]) successfully deleted
I20240502 08:19:03.837991 21745 raft_consensus.cc:2258] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:03.838812 21745 ts_tablet_manager.cc:1875] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:03.842808 21745 ts_tablet_manager.cc:1888] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:19:03.842912 21745 log.cc:1196] T 8cacb00fa8b245a6b8604f90d8f5b618 P 54eba73ef0bb466ab4776d6ea0541b78: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/8cacb00fa8b245a6b8604f90d8f5b618
I20240502 08:19:03.843555 20132 catalog_manager.cc:4830] TS 54eba73ef0bb466ab4776d6ea0541b78 (127.6.151.66:45629): tablet 8cacb00fa8b245a6b8604f90d8f5b618 (table rebalance_test_table_2 [id=b9e518ae71494d64ad8aa253c5d932f3]) successfully deleted
I20240502 08:19:04.133553 20266 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:04.137495 20974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.146183 20399 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.148183 21106 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.296497 20266 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:04.297933 20974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.299585 20399 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.301259 21106 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.594511 20399 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.595742 20266 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:04.598345 20974 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:04.609730 21106 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
               UUID               |       Address       | Status
----------------------------------+---------------------+---------
 9b70339729a74685861bd54114f0b213 | 127.6.151.126:38907 | HEALTHY

Unusual flags for Master:
                   Flag                   |                                                                                               Value                                                                                               |         Tags          |         Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_ca_key_size                         | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 ipki_server_key_size                     | 768                                                                                                                                                                                               | experimental          | all 1 server(s) checked
 never_fsync                              | true                                                                                                                                                                                              | unsafe,advanced       | all 1 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                                 | unsafe,hidden         | all 1 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                              | experimental,advanced | all 1 server(s) checked
 rpc_reuseport                            | true                                                                                                                                                                                              | experimental          | all 1 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                              | unsafe                | all 1 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                                | hidden                | all 1 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden                | all 1 server(s) checked
 tsk_num_rsa_bits                         | 512                                                                                                                                                                                               | experimental          | all 1 server(s) checked

Flags of checked categories for Master:
        Flag         |       Value        |         Master
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:40921 | all 1 server(s) checked
 time_source         | builtin            | all 1 server(s) checked

Tablet Server Summary
               UUID               |      Address       | Status  | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
 21f970fbfa08497f81784d2053e77171 | 127.6.151.65:32959 | HEALTHY | <none>   |       4        |       0
 54eba73ef0bb466ab4776d6ea0541b78 | 127.6.151.66:45629 | HEALTHY | <none>   |       3        |       0
 7c2550bd25424f3f82ad02367f52be2a | 127.6.151.67:36255 | HEALTHY | <none>   |       3        |       0
 f6f542b9bfba42028f61587b9d915542 | 127.6.151.68:34235 | HEALTHY | <none>   |       3        |       0

Tablet Server Location Summary
 Location |  Count
----------+---------
 <none>   |       4

Unusual flags for Tablet Server:
                   Flag                   |                                                                                             Value                                                                                             |         Tags          |      Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
 ipki_server_key_size                     | 768                                                                                                                                                                                           | experimental          | all 4 server(s) checked
 local_ip_for_outbound_sockets            | 127.6.151.65                                                                                                                                                                                  | experimental          | 127.6.151.65:32959
 local_ip_for_outbound_sockets            | 127.6.151.66                                                                                                                                                                                  | experimental          | 127.6.151.66:45629
 local_ip_for_outbound_sockets            | 127.6.151.67                                                                                                                                                                                  | experimental          | 127.6.151.67:36255
 local_ip_for_outbound_sockets            | 127.6.151.68                                                                                                                                                                                  | experimental          | 127.6.151.68:34235
 never_fsync                              | true                                                                                                                                                                                          | unsafe,advanced       | all 4 server(s) checked
 openssl_security_level_override          | 0                                                                                                                                                                                             | unsafe,hidden         | all 4 server(s) checked
 raft_prepare_replacement_before_eviction | true                                                                                                                                                                                          | experimental,advanced | all 4 server(s) checked
 rpc_server_allow_ephemeral_ports         | true                                                                                                                                                                                          | unsafe                | all 4 server(s) checked
 server_dump_info_format                  | pb                                                                                                                                                                                            | hidden                | all 4 server(s) checked
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden                | 127.6.151.65:32959
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden                | 127.6.151.66:45629
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden                | 127.6.151.67:36255
 server_dump_info_path                    | /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_1.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden                | 127.6.151.68:34235

Flags of checked categories for Tablet Server:
        Flag         |       Value        |      Tablet Server
---------------------+--------------------+-------------------------
 builtin_ntp_servers | 127.6.151.84:40921 | all 4 server(s) checked
 time_source         | builtin            | all 4 server(s) checked

Version Summary
     Version     |         Servers
-----------------+-------------------------
 1.18.0-SNAPSHOT | all 5 server(s) checked

Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
          Name          | RF | Status  | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
 TestTable              | 1  | HEALTHY | 1             | 1       | 0          | 0                | 0
 rebalance_test_table_0 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_1 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_2 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0
 rebalance_test_table_3 | 1  | HEALTHY | 3             | 3       | 0          | 0                | 0

Tablet Replica Count Summary
   Statistic    | Replica Count
----------------+---------------
 Minimum        | 3
 First Quartile | 3
 Median         | 3
 Third Quartile | 4
 Maximum        | 4

Total Count Summary
                | Total Count
----------------+-------------
 Masters        | 1
 Tablet Servers | 4
 Tables         | 5
 Tablets        | 13
 Replicas       | 13

==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set

OK
I20240502 08:19:04.664788  6749 log_verifier.cc:126] Checking tablet 00a2296b66cd40f8ae82e8ca2354c065
I20240502 08:19:04.673542  6749 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 00a2296b66cd40f8ae82e8ca2354c065
I20240502 08:19:04.673667  6749 log_verifier.cc:126] Checking tablet 0790c9f2e68e48d7b9513cc6623a348b
I20240502 08:19:04.681243  6749 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0790c9f2e68e48d7b9513cc6623a348b
I20240502 08:19:04.681344  6749 log_verifier.cc:126] Checking tablet 3afda5dcfefa48d5b95a0172847ef938
I20240502 08:19:04.689194  6749 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3afda5dcfefa48d5b95a0172847ef938
I20240502 08:19:04.689296  6749 log_verifier.cc:126] Checking tablet 3eac9a0467f34a3986a0b9cfc61f51e0
I20240502 08:19:04.696759  6749 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3eac9a0467f34a3986a0b9cfc61f51e0
I20240502 08:19:04.696861  6749 log_verifier.cc:126] Checking tablet 3f57a6d216ee4ee6918af6a887a597c3
I20240502 08:19:04.704614  6749 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3f57a6d216ee4ee6918af6a887a597c3
I20240502 08:19:04.704720  6749 log_verifier.cc:126] Checking tablet 50581ab834944158a50bfc5a7c96b57c
I20240502 08:19:04.713197  6749 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 50581ab834944158a50bfc5a7c96b57c
I20240502 08:19:04.713315  6749 log_verifier.cc:126] Checking tablet 711469afa64b4d4dbccaeb20a2e5f483
I20240502 08:19:04.721402  6749 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 711469afa64b4d4dbccaeb20a2e5f483
I20240502 08:19:04.721511  6749 log_verifier.cc:126] Checking tablet 7fd32afde3564645803d716a70853fe1
I20240502 08:19:04.769155  6749 log_verifier.cc:177] Verified matching terms for 5 ops in tablet 7fd32afde3564645803d716a70853fe1
I20240502 08:19:04.769250  6749 log_verifier.cc:126] Checking tablet 8cacb00fa8b245a6b8604f90d8f5b618
I20240502 08:19:04.777451  6749 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8cacb00fa8b245a6b8604f90d8f5b618
I20240502 08:19:04.777549  6749 log_verifier.cc:126] Checking tablet 9fd1e33dc4b4408dac968aa1fa33409d
I20240502 08:19:04.784853  6749 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9fd1e33dc4b4408dac968aa1fa33409d
I20240502 08:19:04.784950  6749 log_verifier.cc:126] Checking tablet b9d36140219745119ae5d6f34d41c66b
I20240502 08:19:04.792264  6749 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b9d36140219745119ae5d6f34d41c66b
I20240502 08:19:04.792351  6749 log_verifier.cc:126] Checking tablet f55c77486e5c42f99f05b43f82269720
I20240502 08:19:04.800804  6749 log_verifier.cc:177] Verified matching terms for 5 ops in tablet f55c77486e5c42f99f05b43f82269720
I20240502 08:19:04.800925  6749 log_verifier.cc:126] Checking tablet f874df79af4c4080bc97fc9ad002b72e
I20240502 08:19:04.810874  6749 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f874df79af4c4080bc97fc9ad002b72e
I20240502 08:19:04.811815  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 20204
I20240502 08:19:04.813117 20328 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:04.932941  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 20204
I20240502 08:19:04.954571  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 20337
I20240502 08:19:04.955695 20459 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:05.071219  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 20337
I20240502 08:19:05.092988  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 20912
I20240502 08:19:05.099602 21033 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:05.205180  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 20912
I20240502 08:19:05.224783  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 21043
I20240502 08:19:05.231408 21165 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:05.340612  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 21043
I20240502 08:19:05.360106  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 20114
I20240502 08:19:05.361297 20175 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:05.489077  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 20114
2024-05-02T08:19:05Z chronyd exiting
[       OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/1 (15330 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (15330 ms total)

[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN      ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3
I20240502 08:19:05.546697  6749 test_util.cc:268] Using random seed: 1101637513
I20240502 08:19:05.548492  6749 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:19:05.548563  6749 ts_itest-base.cc:113] --------------
I20240502 08:19:05.548629  6749 ts_itest-base.cc:114] 6 tablet servers
I20240502 08:19:05.548676  6749 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:19:05.548755  6749 ts_itest-base.cc:116] --------------
2024-05-02T08:19:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:19:05Z Disabled control of system clock
I20240502 08:19:05.601972  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:44205
--webserver_interface=127.6.151.126
--webserver_port=0
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.151.126:44205
--location_mapping_cmd=/tmp/dist-test-taskOUdoRy/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 08:19:05.945508 21899 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:19:05.945897 21899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:05.946012 21899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:05.946125 21899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:05.953878 21899 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:05.954038 21899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:19:05.954160 21899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:05.954229 21899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:19:05.954334 21899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:19:05.963734 21899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskOUdoRy/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:3
--ipki_ca_key_size=768
--master_addresses=127.6.151.126:44205
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.151.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:05.965128 21899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:05.966950 21899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:05.978206 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:05.978708 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:05.979326 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:05.980104 21899 server_base.cc:1034] running on GCE node
I20240502 08:19:05.980883 21899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:05.982353 21899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:05.983589 21899 hybrid_clock.cc:648] HybridClock initialized: now 1714637945983517 us; error 54 us; skew 500 ppm
I20240502 08:19:05.983907 21899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:05.990788 21899 webserver.cc:460] Webserver started at http://127.6.151.126:45833/ using document root <none> and password file <none>
I20240502 08:19:05.991811 21899 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:05.991991 21899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:05.992380 21899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:05.997220 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "81356d8e20284f4d8af2119064867053"
format_stamp: "Formatted at 2024-05-02 08:19:05 on dist-test-slave-x3qn"
I20240502 08:19:05.998046 21899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "81356d8e20284f4d8af2119064867053"
format_stamp: "Formatted at 2024-05-02 08:19:05 on dist-test-slave-x3qn"
I20240502 08:19:06.003643 21899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20240502 08:19:06.007139 21914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:06.008565 21899 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 08:19:06.008728 21899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
uuid: "81356d8e20284f4d8af2119064867053"
format_stamp: "Formatted at 2024-05-02 08:19:05 on dist-test-slave-x3qn"
I20240502 08:19:06.008899 21899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:06.027976 21899 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:06.028930 21899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:06.029218 21899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:06.048182 21899 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.126:44205
I20240502 08:19:06.048213 21965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.126:44205 every 8 connection(s)
I20240502 08:19:06.049595 21899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:19:06.053347  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 21899
I20240502 08:19:06.053534 21966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:06.053586  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:19:06.066514 21966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053: Bootstrap starting.
I20240502 08:19:06.069571 21966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:06.070590 21966 log.cc:824] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:06.073357 21966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053: No bootstrap required, opened a new log
I20240502 08:19:06.079658 21966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81356d8e20284f4d8af2119064867053" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 44205 } }
I20240502 08:19:06.079996 21966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:06.080103 21966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81356d8e20284f4d8af2119064867053, State: Initialized, Role: FOLLOWER
I20240502 08:19:06.080686 21966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81356d8e20284f4d8af2119064867053" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 44205 } }
I20240502 08:19:06.080885 21966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:19:06.081014 21966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:19:06.081169 21966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:06.083510 21966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81356d8e20284f4d8af2119064867053" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 44205 } }
I20240502 08:19:06.083984 21966 leader_election.cc:302] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81356d8e20284f4d8af2119064867053; no voters: 
I20240502 08:19:06.084575 21966 leader_election.cc:288] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:19:06.084702 21971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:06.085381 21971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [term 1 LEADER]: Becoming Leader. State: Replica: 81356d8e20284f4d8af2119064867053, State: Running, Role: LEADER
I20240502 08:19:06.085983 21971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81356d8e20284f4d8af2119064867053" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 44205 } }
I20240502 08:19:06.086553 21966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:19:06.089051 21972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81356d8e20284f4d8af2119064867053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81356d8e20284f4d8af2119064867053" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 44205 } } }
I20240502 08:19:06.089159 21973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81356d8e20284f4d8af2119064867053. Latest consensus state: current_term: 1 leader_uuid: "81356d8e20284f4d8af2119064867053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81356d8e20284f4d8af2119064867053" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 44205 } } }
I20240502 08:19:06.089438 21972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [sys.catalog]: This master's current role is: LEADER
I20240502 08:19:06.089555 21973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053 [sys.catalog]: This master's current role is: LEADER
I20240502 08:19:06.090826 21977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:19:06.096040 21977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:19:06.104595 21977 catalog_manager.cc:1347] Generated new cluster ID: 5c56dbb5794f49a4983fc9f3a771b28d
I20240502 08:19:06.104736 21977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:19:06.123032 21977 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:19:06.124439 21977 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:19:06.136334 21977 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 81356d8e20284f4d8af2119064867053: Generated new TSK 0
I20240502 08:19:06.137714 21977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:19:06.153983  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.65:0
--local_ip_for_outbound_sockets=127.6.151.65
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:19:06.503418 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:06.503710 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:06.503865 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:06.511389 21990 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:06.511515 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:06.511677 21990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.65
I20240502 08:19:06.520913 21990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.65
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:06.522238 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:06.523942 21990 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:06.536070 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:06.536628 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:06.536945 21998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:06.537629 21990 server_base.cc:1034] running on GCE node
I20240502 08:19:06.538237 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:06.539192 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:06.540405 21990 hybrid_clock.cc:648] HybridClock initialized: now 1714637946540358 us; error 38 us; skew 500 ppm
I20240502 08:19:06.540714 21990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:06.543021 21990 webserver.cc:460] Webserver started at http://127.6.151.65:35379/ using document root <none> and password file <none>
I20240502 08:19:06.543736 21990 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:06.543866 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:06.544165 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:06.546968 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
format_stamp: "Formatted at 2024-05-02 08:19:06 on dist-test-slave-x3qn"
I20240502 08:19:06.547719 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
format_stamp: "Formatted at 2024-05-02 08:19:06 on dist-test-slave-x3qn"
I20240502 08:19:06.552834 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:19:06.556145 22004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:06.557540 21990 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:19:06.557770 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
format_stamp: "Formatted at 2024-05-02 08:19:06 on dist-test-slave-x3qn"
I20240502 08:19:06.557977 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:06.576224 21990 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:06.577093 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:06.577370 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:06.578342 21990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:06.579914 21990 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:06.580013 21990 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:06.580142 21990 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:06.580232 21990 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:06.612787 21990 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.65:45747
I20240502 08:19:06.612955 22117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.65:45747 every 8 connection(s)
I20240502 08:19:06.614147 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:19:06.623323  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 21990
I20240502 08:19:06.623529  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:19:06.627111  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.66:0
--local_ip_for_outbound_sockets=127.6.151.66
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:06.627808 22118 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:06.628132 22118 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:06.628971 22118 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:06.710217 21931 ts_manager.cc:194] Registered new tserver with Master: d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747)
I20240502 08:19:06.712121 21931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.65:50869
W20240502 08:19:07.013105 22123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:07.013412 22123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:07.013520 22123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:07.021888 22123 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:07.022018 22123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:07.022156 22123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.66
I20240502 08:19:07.031771 22123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.66
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:07.033084 22123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:07.035034 22123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:07.046734 22129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:07.047343 22128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:07.047569 22131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:07.048696 22123 server_base.cc:1034] running on GCE node
I20240502 08:19:07.049263 22123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:07.050251 22123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:07.051470 22123 hybrid_clock.cc:648] HybridClock initialized: now 1714637947051426 us; error 43 us; skew 500 ppm
I20240502 08:19:07.051813 22123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:07.054059 22123 webserver.cc:460] Webserver started at http://127.6.151.66:35337/ using document root <none> and password file <none>
I20240502 08:19:07.054832 22123 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:07.054957 22123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:07.055251 22123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:07.058017 22123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8ef0b298bd6642ea900f3e9843bc3031"
format_stamp: "Formatted at 2024-05-02 08:19:07 on dist-test-slave-x3qn"
I20240502 08:19:07.058992 22123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8ef0b298bd6642ea900f3e9843bc3031"
format_stamp: "Formatted at 2024-05-02 08:19:07 on dist-test-slave-x3qn"
I20240502 08:19:07.064529 22123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:19:07.068127 22137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:07.069542 22123 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
I20240502 08:19:07.069723 22123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8ef0b298bd6642ea900f3e9843bc3031"
format_stamp: "Formatted at 2024-05-02 08:19:07 on dist-test-slave-x3qn"
I20240502 08:19:07.069947 22123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:07.093319 22123 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:07.094208 22123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:07.094480 22123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:07.095501 22123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:07.097059 22123 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:07.097165 22123 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:07.097280 22123 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:07.097364 22123 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:07.128165 22123 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.66:42597
I20240502 08:19:07.128221 22250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.66:42597 every 8 connection(s)
I20240502 08:19:07.129515 22123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:19:07.129792  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 22123
I20240502 08:19:07.129956  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:19:07.134581  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:0
--local_ip_for_outbound_sockets=127.6.151.67
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:07.144680 22251 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:07.145064 22251 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:07.146327 22251 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:07.195865 21931 ts_manager.cc:194] Registered new tserver with Master: 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:07.197153 21931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.66:52357
W20240502 08:19:07.481339 22255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:07.481637 22255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:07.481725 22255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:07.489259 22255 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:07.489385 22255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:07.489512 22255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:19:07.498611 22255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:07.499953 22255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:07.501607 22255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:07.513342 22261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:07.513908 22262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:07.514564 22264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:07.515275 22255 server_base.cc:1034] running on GCE node
I20240502 08:19:07.515856 22255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:07.516752 22255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:07.517998 22255 hybrid_clock.cc:648] HybridClock initialized: now 1714637947517951 us; error 46 us; skew 500 ppm
I20240502 08:19:07.518308 22255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:07.520529 22255 webserver.cc:460] Webserver started at http://127.6.151.67:41331/ using document root <none> and password file <none>
I20240502 08:19:07.521315 22255 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:07.521440 22255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:07.521742 22255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:07.524369 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3"
format_stamp: "Formatted at 2024-05-02 08:19:07 on dist-test-slave-x3qn"
I20240502 08:19:07.525138 22255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3"
format_stamp: "Formatted at 2024-05-02 08:19:07 on dist-test-slave-x3qn"
I20240502 08:19:07.530326 22255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20240502 08:19:07.533872 22270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:07.535308 22255 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20240502 08:19:07.535463 22255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3"
format_stamp: "Formatted at 2024-05-02 08:19:07 on dist-test-slave-x3qn"
I20240502 08:19:07.535658 22255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:07.572165 22255 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:07.573035 22255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:07.573314 22255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:07.574353 22255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:07.575917 22255 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:07.576016 22255 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:07.576136 22255 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:07.576221 22255 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:07.608932 22255 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:41709
I20240502 08:19:07.609021 22383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:41709 every 8 connection(s)
I20240502 08:19:07.610246 22255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:19:07.612993  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 22255
I20240502 08:19:07.613221  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:19:07.616632  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:0
--local_ip_for_outbound_sockets=127.6.151.68
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:07.625437 22384 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:07.625761 22384 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:07.626713 22384 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:07.688185 21931 ts_manager.cc:194] Registered new tserver with Master: ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:07.689160 21931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:41279
I20240502 08:19:07.715433 22118 heartbeater.cc:503] Master 127.6.151.126:44205 was elected leader, sending a full tablet report...
W20240502 08:19:07.978614 22387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:07.978972 22387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:07.979120 22387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:07.986629 22387 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:07.986800 22387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:07.986968 22387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:19:07.996169 22387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:07.997925 22387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:07.999927 22387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:08.012077 22393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:08.012352 22396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:08.012957 22394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:08.013655 22387 server_base.cc:1034] running on GCE node
I20240502 08:19:08.014261 22387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:08.015251 22387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:08.016453 22387 hybrid_clock.cc:648] HybridClock initialized: now 1714637948016399 us; error 42 us; skew 500 ppm
I20240502 08:19:08.016727 22387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:08.018957 22387 webserver.cc:460] Webserver started at http://127.6.151.68:38779/ using document root <none> and password file <none>
I20240502 08:19:08.019631 22387 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:08.019734 22387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:08.020011 22387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:08.022445 22387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.023190 22387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.028120 22387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.000s	sys 0.004s
I20240502 08:19:08.031370 22402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:08.032737 22387 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:19:08.032910 22387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.033097 22387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:08.055780 22387 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:08.056635 22387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:08.056912 22387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:08.057899 22387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:08.059415 22387 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:08.059513 22387 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:08.059635 22387 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:08.059710 22387 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:08.091075 22387 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:34181
I20240502 08:19:08.091130 22515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:34181 every 8 connection(s)
I20240502 08:19:08.092397 22387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:19:08.094712  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 22387
I20240502 08:19:08.094882  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:19:08.098433  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:0
--local_ip_for_outbound_sockets=127.6.151.69
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:08.108000 22516 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:08.108376 22516 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:08.109683 22516 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:08.160632 21931 ts_manager.cc:194] Registered new tserver with Master: b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181)
I20240502 08:19:08.161792 21931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:54507
I20240502 08:19:08.200314 22251 heartbeater.cc:503] Master 127.6.151.126:44205 was elected leader, sending a full tablet report...
W20240502 08:19:08.453498 22519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:08.453855 22519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:08.454043 22519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:08.461922 22519 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:08.462040 22519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:08.462194 22519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:19:08.471300 22519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:08.472602 22519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:08.474360 22519 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:08.487368 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:08.487980 22526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:08.488509 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:08.489421 22519 server_base.cc:1034] running on GCE node
I20240502 08:19:08.490025 22519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:08.491052 22519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:08.492283 22519 hybrid_clock.cc:648] HybridClock initialized: now 1714637948492239 us; error 34 us; skew 500 ppm
I20240502 08:19:08.492645 22519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:08.495057 22519 webserver.cc:460] Webserver started at http://127.6.151.69:45493/ using document root <none> and password file <none>
I20240502 08:19:08.495811 22519 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:08.495927 22519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:08.496241 22519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:08.498932 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "26d5ad8a5f594649b73ab2b11dc3d328"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.499722 22519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "26d5ad8a5f594649b73ab2b11dc3d328"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.504945 22519 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:19:08.508386 22534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:08.509733 22519 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:19:08.509910 22519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "26d5ad8a5f594649b73ab2b11dc3d328"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.510118 22519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:08.533831 22519 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:08.534763 22519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:08.535037 22519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:08.536089 22519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:08.537745 22519 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:08.537843 22519 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:08.537963 22519 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:08.538091 22519 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20240502 08:19:08.569559 22519 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:41139
I20240502 08:19:08.569618 22647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:41139 every 8 connection(s)
I20240502 08:19:08.570910 22519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:19:08.572875  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 22519
I20240502 08:19:08.573096  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:19:08.576830  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.70:0
--local_ip_for_outbound_sockets=127.6.151.70
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:08.586251 22648 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:08.586653 22648 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:08.587736 22648 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:08.640944 21931 ts_manager.cc:194] Registered new tserver with Master: 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69:41139)
I20240502 08:19:08.642158 21931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:41721
I20240502 08:19:08.692516 22384 heartbeater.cc:503] Master 127.6.151.126:44205 was elected leader, sending a full tablet report...
W20240502 08:19:08.933171 22651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:08.933490 22651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:08.933634 22651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:08.941676 22651 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:08.941835 22651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:08.941998 22651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.70
I20240502 08:19:08.951400 22651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.151.70
--webserver_port=0
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.70
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:08.952788 22651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:08.954520 22651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:08.966780 22657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:08.967376 22658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:08.967998 22660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:08.968611 22651 server_base.cc:1034] running on GCE node
I20240502 08:19:08.969178 22651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:08.970135 22651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:08.971338 22651 hybrid_clock.cc:648] HybridClock initialized: now 1714637948971283 us; error 36 us; skew 500 ppm
I20240502 08:19:08.971674 22651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:08.973906 22651 webserver.cc:460] Webserver started at http://127.6.151.70:33159/ using document root <none> and password file <none>
I20240502 08:19:08.974712 22651 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:08.974825 22651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:08.975106 22651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:08.977767 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fdea103695a94a099acd05c42bcccdca"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.978544 22651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fdea103695a94a099acd05c42bcccdca"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.983762 22651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:19:08.987219 22666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:08.988596 22651 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20240502 08:19:08.988735 22651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fdea103695a94a099acd05c42bcccdca"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:08.988912 22651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:09.015136 22651 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:09.016009 22651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:09.016253 22651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:09.017176 22651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:09.018695 22651 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:09.018795 22651 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:09.018887 22651 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:09.018944 22651 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:09.051812 22651 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.70:46149
I20240502 08:19:09.051852 22779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.70:46149 every 8 connection(s)
I20240502 08:19:09.053177 22651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:19:09.063395  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 22651
I20240502 08:19:09.063567  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 08:19:09.066180 22780 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:09.066516 22780 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:09.067648 22780 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:09.114595 21931 ts_manager.cc:194] Registered new tserver with Master: fdea103695a94a099acd05c42bcccdca (127.6.151.70:46149)
I20240502 08:19:09.115598 21931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.70:37603
I20240502 08:19:09.119634  6749 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 08:19:09.136893  6749 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240502 08:19:09.137053  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 22651
I20240502 08:19:09.145630 22775 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:09.164693 22516 heartbeater.cc:503] Master 127.6.151.126:44205 was elected leader, sending a full tablet report...
I20240502 08:19:09.229122  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 22651
I20240502 08:19:09.249238  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 22519
I20240502 08:19:09.257342 22643 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:09.339977  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 22519
I20240502 08:19:09.357563  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 22387
I20240502 08:19:09.366142 22511 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:09.447204  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 22387
I20240502 08:19:13.215499 21931 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38388:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 08:19:13.222138 21931 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:19:13.271601 22316 tablet_service.cc:1467] Processing CreateTablet for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.273061 22316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.275022 22318 tablet_service.cc:1467] Processing CreateTablet for tablet 5507342ac56b42878ce02a2dcf149bc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.275584 22318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5507342ac56b42878ce02a2dcf149bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.275202 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 82553085391e47fdae649aa372f54f37 (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.276175 22183 tablet_service.cc:1467] Processing CreateTablet for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.274971 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 5507342ac56b42878ce02a2dcf149bc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.276664 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82553085391e47fdae649aa372f54f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.277208 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.278841 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5507342ac56b42878ce02a2dcf149bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.278928 22317 tablet_service.cc:1467] Processing CreateTablet for tablet 82553085391e47fdae649aa372f54f37 (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.279356 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82553085391e47fdae649aa372f54f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.293356 22050 tablet_service.cc:1467] Processing CreateTablet for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.296167 22051 tablet_service.cc:1467] Processing CreateTablet for tablet 82553085391e47fdae649aa372f54f37 (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.297446 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.302878 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82553085391e47fdae649aa372f54f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.297951 22052 tablet_service.cc:1467] Processing CreateTablet for tablet 5507342ac56b42878ce02a2dcf149bc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:13.304160 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5507342ac56b42878ce02a2dcf149bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:13.307567 22807 tablet_bootstrap.cc:492] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:13.310370 22807 tablet_bootstrap.cc:654] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.311703 22807 log.cc:824] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:13.325991 22809 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:13.326162 22806 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:13.327136 22807 tablet_bootstrap.cc:492] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:13.327422 22807 ts_tablet_manager.cc:1367] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.010s
I20240502 08:19:13.329980 22809 tablet_bootstrap.cc:654] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.329983 22806 tablet_bootstrap.cc:654] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.331331 22809 log.cc:824] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:13.331334 22806 log.cc:824] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:13.333882 22806 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:13.334137 22806 ts_tablet_manager.cc:1367] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.008s	user 0.006s	sys 0.000s
I20240502 08:19:13.334344 22809 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:13.334614 22809 ts_tablet_manager.cc:1367] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.009s	user 0.006s	sys 0.000s
I20240502 08:19:13.334807 22807 raft_consensus.cc:348] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.335268 22807 raft_consensus.cc:374] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.335435 22807 raft_consensus.cc:729] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.336206 22807 consensus_queue.cc:260] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.337982 22807 ts_tablet_manager.cc:1398] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20240502 08:19:13.338423 22807 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:13.340137 22807 tablet_bootstrap.cc:654] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.343429 22806 raft_consensus.cc:348] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.344009 22806 raft_consensus.cc:374] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.344169 22806 raft_consensus.cc:729] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.344090 22809 raft_consensus.cc:348] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.344611 22809 raft_consensus.cc:374] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.344777 22809 raft_consensus.cc:729] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.344971 22806 consensus_queue.cc:260] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.345571 22809 consensus_queue.cc:260] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.347010 22806 ts_tablet_manager.cc:1398] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.004s
I20240502 08:19:13.347466 22806 tablet_bootstrap.cc:492] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:13.349320 22806 tablet_bootstrap.cc:654] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.349687 22807 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:13.349946 22807 ts_tablet_manager.cc:1367] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.012s	user 0.010s	sys 0.000s
I20240502 08:19:13.351670 22807 raft_consensus.cc:348] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.352031 22807 raft_consensus.cc:374] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.352193 22807 raft_consensus.cc:729] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.352613 22807 consensus_queue.cc:260] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.353411 22807 ts_tablet_manager.cc:1398] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:13.353847 22807 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:13.356213 22807 tablet_bootstrap.cc:654] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.357626 22806 tablet_bootstrap.cc:492] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:13.357896 22806 ts_tablet_manager.cc:1367] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.007s
I20240502 08:19:13.359320 22809 ts_tablet_manager.cc:1398] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.025s	user 0.016s	sys 0.006s
I20240502 08:19:13.359959 22809 tablet_bootstrap.cc:492] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:13.360278 22806 raft_consensus.cc:348] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.360605 22806 raft_consensus.cc:374] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.360759 22806 raft_consensus.cc:729] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.361122 22806 consensus_queue.cc:260] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.361639 22807 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:13.361689 22806 ts_tablet_manager.cc:1398] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.004s	user 0.001s	sys 0.001s
I20240502 08:19:13.361862 22807 ts_tablet_manager.cc:1367] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 08:19:13.361939 22809 tablet_bootstrap.cc:654] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.362053 22806 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:13.363914 22806 tablet_bootstrap.cc:654] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.363713 22807 raft_consensus.cc:348] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.364022 22807 raft_consensus.cc:374] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.364181 22807 raft_consensus.cc:729] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.364638 22807 consensus_queue.cc:260] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.365442 22807 ts_tablet_manager.cc:1398] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
W20240502 08:19:13.368317 22385 tablet.cc:2369] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:19:13.370414 22809 tablet_bootstrap.cc:492] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:13.370661 22809 ts_tablet_manager.cc:1367] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.007s
W20240502 08:19:13.371306 22119 tablet.cc:2369] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:19:13.371690 22809 raft_consensus.cc:348] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.371886 22806 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:13.371908 22809 raft_consensus.cc:374] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.372023 22809 raft_consensus.cc:729] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.372071 22806 ts_tablet_manager.cc:1367] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.010s	user 0.003s	sys 0.007s
I20240502 08:19:13.372248 22809 consensus_queue.cc:260] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.372736 22809 ts_tablet_manager.cc:1398] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:13.372723 22806 raft_consensus.cc:348] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.372968 22806 raft_consensus.cc:374] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.373009 22809 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:13.373055 22806 raft_consensus.cc:729] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.373310 22806 consensus_queue.cc:260] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.374017 22806 ts_tablet_manager.cc:1398] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:19:13.374579 22809 tablet_bootstrap.cc:654] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:13.376762 22809 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:13.376909 22809 ts_tablet_manager.cc:1367] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.000s
I20240502 08:19:13.377410 22809 raft_consensus.cc:348] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.377611 22809 raft_consensus.cc:374] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:13.377717 22809 raft_consensus.cc:729] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:13.377949 22809 consensus_queue.cc:260] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.378404 22809 ts_tablet_manager.cc:1398] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
W20240502 08:19:13.386056 22252 tablet.cc:2369] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:19:13.491391 22814 raft_consensus.cc:482] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:13.491693 22814 raft_consensus.cc:504] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.493523 22814 leader_election.cc:288] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:13.500597 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3b3b39b7b54ef3b7c3b5e7265b76de" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" is_pre_election: true
I20240502 08:19:13.500597 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3b3b39b7b54ef3b7c3b5e7265b76de" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" is_pre_election: true
I20240502 08:19:13.501116 22072 raft_consensus.cc:2454] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 0.
I20240502 08:19:13.501118 22338 raft_consensus.cc:2454] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 0.
I20240502 08:19:13.501693 22141 leader_election.cc:302] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, d24c3c61ac4a4eefa0e4a5d9cef079b8; no voters: 
I20240502 08:19:13.502067 22814 raft_consensus.cc:2781] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:13.502193 22814 raft_consensus.cc:482] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:13.502315 22814 raft_consensus.cc:3037] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.504412 22814 raft_consensus.cc:504] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.504987 22814 leader_election.cc:288] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 election: Requested vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:13.505430 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3b3b39b7b54ef3b7c3b5e7265b76de" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
I20240502 08:19:13.505515 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd3b3b39b7b54ef3b7c3b5e7265b76de" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3"
I20240502 08:19:13.505653 22072 raft_consensus.cc:3037] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.505707 22338 raft_consensus.cc:3037] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.508782 22338 raft_consensus.cc:2454] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 1.
I20240502 08:19:13.508782 22072 raft_consensus.cc:2454] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 1.
I20240502 08:19:13.509229 22139 leader_election.cc:302] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:13.509512 22814 raft_consensus.cc:2781] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:13.509958 22814 raft_consensus.cc:686] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Running, Role: LEADER
I20240502 08:19:13.510413 22814 consensus_queue.cc:237] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.515383 21930 catalog_manager.cc:5484] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:13.522437 22812 raft_consensus.cc:482] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:13.522625 22812 raft_consensus.cc:504] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.524299 22812 leader_election.cc:288] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:13.531064 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82553085391e47fdae649aa372f54f37" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" is_pre_election: true
I20240502 08:19:13.531318 22072 raft_consensus.cc:2454] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:13.531563 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82553085391e47fdae649aa372f54f37" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031" is_pre_election: true
I20240502 08:19:13.531774 22274 leader_election.cc:302] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:13.531915 22206 raft_consensus.cc:2454] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:13.532171 22812 raft_consensus.cc:2781] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:13.532286 22812 raft_consensus.cc:482] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:13.532393 22812 raft_consensus.cc:3037] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.534397 22812 raft_consensus.cc:504] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.535012 22812 leader_election.cc:288] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Requested vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:13.535439 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82553085391e47fdae649aa372f54f37" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
I20240502 08:19:13.535521 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82553085391e47fdae649aa372f54f37" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031"
I20240502 08:19:13.535629 22072 raft_consensus.cc:3037] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.535718 22206 raft_consensus.cc:3037] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.537482 22072 raft_consensus.cc:2454] T 82553085391e47fdae649aa372f54f37 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:13.537730 22206 raft_consensus.cc:2454] T 82553085391e47fdae649aa372f54f37 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:13.537874 22274 leader_election.cc:302] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:13.538178 22812 raft_consensus.cc:2781] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:13.538586 22812 raft_consensus.cc:686] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Becoming Leader. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Running, Role: LEADER
I20240502 08:19:13.539149 22812 consensus_queue.cc:237] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:13.543749 21930 catalog_manager.cc:5484] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67). New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:13.595963 22815 raft_consensus.cc:482] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:13.596190 22815 raft_consensus.cc:504] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.597872 22815 leader_election.cc:288] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:13.604777 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5507342ac56b42878ce02a2dcf149bc0" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031" is_pre_election: true
I20240502 08:19:13.604873 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5507342ac56b42878ce02a2dcf149bc0" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" is_pre_election: true
I20240502 08:19:13.605062 22206 raft_consensus.cc:2454] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 0.
I20240502 08:19:13.605119 22338 raft_consensus.cc:2454] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 0.
I20240502 08:19:13.605573 22006 leader_election.cc:302] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:13.605984 22815 raft_consensus.cc:2781] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:13.606097 22815 raft_consensus.cc:482] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:13.606200 22815 raft_consensus.cc:3037] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.608206 22815 raft_consensus.cc:504] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.608772 22815 leader_election.cc:288] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 election: Requested vote from peers ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:13.609210 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5507342ac56b42878ce02a2dcf149bc0" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031"
I20240502 08:19:13.609269 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5507342ac56b42878ce02a2dcf149bc0" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3"
I20240502 08:19:13.609411 22206 raft_consensus.cc:3037] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.609500 22338 raft_consensus.cc:3037] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:13.611927 22206 raft_consensus.cc:2454] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 1.
I20240502 08:19:13.612277 22338 raft_consensus.cc:2454] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 1.
I20240502 08:19:13.612319 22008 leader_election.cc:302] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, d24c3c61ac4a4eefa0e4a5d9cef079b8; no voters: 
I20240502 08:19:13.612672 22815 raft_consensus.cc:2781] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:13.614900 22815 raft_consensus.cc:686] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Becoming Leader. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Running, Role: LEADER
I20240502 08:19:13.615540 22815 consensus_queue.cc:237] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } }
I20240502 08:19:13.620668 21930 catalog_manager.cc:5484] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:14.005875 22814 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:14.013854 22823 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:14.048218 22833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:14.048579 22833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:19:14.055755 22812 consensus_queue.cc:1035] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:14.057543 22833 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:19:14.064994 22812 consensus_queue.cc:1035] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:14.088480 22815 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:14.093762 22831 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'

INSERT report
    rows total: 2000
    time total: 243.468 ms
  time per row: 0.121734 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
     18       4816 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:19:15.372902 21929 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38388:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 08:19:15.373692 21929 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:19:15.382149 22183 tablet_service.cc:1467] Processing CreateTablet for tablet 82db7fca82b5492faa762066294de080 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.382472 22317 tablet_service.cc:1467] Processing CreateTablet for tablet 82db7fca82b5492faa762066294de080 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.382681 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82db7fca82b5492faa762066294de080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.382932 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82db7fca82b5492faa762066294de080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.382905 22050 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd539305b35444596c668a83ace2fbd (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.383009 22052 tablet_service.cc:1467] Processing CreateTablet for tablet 82db7fca82b5492faa762066294de080 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.383293 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd539305b35444596c668a83ace2fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.383656 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82db7fca82b5492faa762066294de080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.384336 22316 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd539305b35444596c668a83ace2fbd (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.384717 22316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd539305b35444596c668a83ace2fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.390762 22318 tablet_service.cc:1467] Processing CreateTablet for tablet 8069f943c7234829b59c930515491ad5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.391196 22318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8069f943c7234829b59c930515491ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.391218 22902 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:15.393990 22902 tablet_bootstrap.cc:654] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.394224 22901 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:15.395234 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 4cd539305b35444596c668a83ace2fbd (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.395684 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd539305b35444596c668a83ace2fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.396442 22901 tablet_bootstrap.cc:654] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.398995 22901 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:15.399251 22901 ts_tablet_manager.cc:1367] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:19:15.400297 22901 raft_consensus.cc:348] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.400427 22051 tablet_service.cc:1467] Processing CreateTablet for tablet 8069f943c7234829b59c930515491ad5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.400719 22901 raft_consensus.cc:374] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.400831 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8069f943c7234829b59c930515491ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.400882 22901 raft_consensus.cc:729] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.401201 22901 consensus_queue.cc:260] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.401999 22901 ts_tablet_manager.cc:1398] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:15.402384 22903 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:15.402529 22183 tablet_service.cc:1467] Processing CreateTablet for tablet 8069f943c7234829b59c930515491ad5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:15.402909 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8069f943c7234829b59c930515491ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:15.404047 22903 tablet_bootstrap.cc:654] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.413424 22903 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:15.413794 22903 ts_tablet_manager.cc:1367] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.011s	user 0.006s	sys 0.003s
I20240502 08:19:15.415329 22901 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:15.416481 22903 raft_consensus.cc:348] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.416816 22903 raft_consensus.cc:374] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.416989 22903 raft_consensus.cc:729] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.417349 22903 consensus_queue.cc:260] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.418494 22901 tablet_bootstrap.cc:654] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.419947 22902 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:15.420213 22902 ts_tablet_manager.cc:1367] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.008s
I20240502 08:19:15.421379 22902 raft_consensus.cc:348] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.421752 22902 raft_consensus.cc:374] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.421900 22902 raft_consensus.cc:729] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.422173 22902 consensus_queue.cc:260] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.422456 22901 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:15.422626 22901 ts_tablet_manager.cc:1367] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 08:19:15.422844 22903 ts_tablet_manager.cc:1398] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20240502 08:19:15.422896 22902 ts_tablet_manager.cc:1398] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:19:15.423139 22903 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:15.423158 22902 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:15.423508 22901 raft_consensus.cc:348] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.423753 22901 raft_consensus.cc:374] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.423859 22901 raft_consensus.cc:729] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.424121 22901 consensus_queue.cc:260] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.424767 22901 ts_tablet_manager.cc:1398] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:19:15.424897 22903 tablet_bootstrap.cc:654] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.425123 22901 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:15.425621 22902 tablet_bootstrap.cc:654] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.426921 22901 tablet_bootstrap.cc:654] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.433208 22903 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:15.433511 22903 ts_tablet_manager.cc:1367] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.010s	user 0.006s	sys 0.003s
I20240502 08:19:15.434453 22902 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:15.434386 22903 raft_consensus.cc:348] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.434705 22902 ts_tablet_manager.cc:1367] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.012s	user 0.006s	sys 0.003s
I20240502 08:19:15.434703 22903 raft_consensus.cc:374] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.434789 22903 raft_consensus.cc:729] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.435135 22903 consensus_queue.cc:260] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.435789 22903 ts_tablet_manager.cc:1398] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:15.435683 22902 raft_consensus.cc:348] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.435901 22902 raft_consensus.cc:374] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.436013 22902 raft_consensus.cc:729] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.436196 22903 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:15.436398 22902 consensus_queue.cc:260] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.437147 22902 ts_tablet_manager.cc:1398] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:19:15.437512 22902 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:15.437907 22903 tablet_bootstrap.cc:654] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.439390 22902 tablet_bootstrap.cc:654] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:15.439839 22901 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:15.440022 22901 ts_tablet_manager.cc:1367] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.008s
I20240502 08:19:15.440975 22901 raft_consensus.cc:348] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.441337 22901 raft_consensus.cc:374] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.441496 22901 raft_consensus.cc:729] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.441527 22902 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:15.441802 22902 ts_tablet_manager.cc:1367] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.001s
I20240502 08:19:15.441926 22901 consensus_queue.cc:260] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.442760 22901 ts_tablet_manager.cc:1398] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:19:15.442667 22902 raft_consensus.cc:348] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.442870 22902 raft_consensus.cc:374] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.443014 22902 raft_consensus.cc:729] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.443272 22902 consensus_queue.cc:260] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.443989 22902 ts_tablet_manager.cc:1398] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:15.447510 22903 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:15.447646 22903 ts_tablet_manager.cc:1367] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.012s	user 0.007s	sys 0.000s
I20240502 08:19:15.448310 22903 raft_consensus.cc:348] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.448462 22903 raft_consensus.cc:374] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:15.448554 22903 raft_consensus.cc:729] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:15.448757 22903 consensus_queue.cc:260] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.449168 22903 ts_tablet_manager.cc:1398] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:15.541741 22814 raft_consensus.cc:482] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:15.541954 22814 raft_consensus.cc:504] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.542615 22814 leader_election.cc:288] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:15.543316 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd539305b35444596c668a83ace2fbd" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" is_pre_election: true
I20240502 08:19:15.543316 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd539305b35444596c668a83ace2fbd" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" is_pre_election: true
I20240502 08:19:15.543568 22072 raft_consensus.cc:2454] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 0.
I20240502 08:19:15.543568 22338 raft_consensus.cc:2454] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 0.
I20240502 08:19:15.544060 22139 leader_election.cc:302] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:15.544370 22814 raft_consensus.cc:2781] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:15.544473 22814 raft_consensus.cc:482] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:15.544569 22814 raft_consensus.cc:3037] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.546439 22814 raft_consensus.cc:504] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.547039 22814 leader_election.cc:288] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 election: Requested vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:15.547513 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd539305b35444596c668a83ace2fbd" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3"
I20240502 08:19:15.547518 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cd539305b35444596c668a83ace2fbd" candidate_uuid: "8ef0b298bd6642ea900f3e9843bc3031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
I20240502 08:19:15.547698 22338 raft_consensus.cc:3037] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.547683 22072 raft_consensus.cc:3037] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.549589 22338 raft_consensus.cc:2454] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 1.
I20240502 08:19:15.549907 22072 raft_consensus.cc:2454] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ef0b298bd6642ea900f3e9843bc3031 in term 1.
I20240502 08:19:15.550083 22139 leader_election.cc:302] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:15.550370 22814 raft_consensus.cc:2781] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:15.550791 22814 raft_consensus.cc:686] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Becoming Leader. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Running, Role: LEADER
I20240502 08:19:15.551064 22814 consensus_queue.cc:237] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.553273 22891 raft_consensus.cc:482] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:15.553396 22891 raft_consensus.cc:504] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.553969 22891 leader_election.cc:288] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:15.553788 21925 catalog_manager.cc:5484] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:15.554476 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82db7fca82b5492faa762066294de080" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031" is_pre_election: true
I20240502 08:19:15.554502 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82db7fca82b5492faa762066294de080" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" is_pre_election: true
I20240502 08:19:15.554688 22206 raft_consensus.cc:2454] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:15.554734 22072 raft_consensus.cc:2454] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:15.555150 22274 leader_election.cc:302] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:15.555408 22891 raft_consensus.cc:2781] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:15.555536 22891 raft_consensus.cc:482] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:15.555629 22891 raft_consensus.cc:3037] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.557869 22891 raft_consensus.cc:504] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.558406 22891 leader_election.cc:288] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Requested vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:15.558789 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82db7fca82b5492faa762066294de080" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
I20240502 08:19:15.558876 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82db7fca82b5492faa762066294de080" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031"
I20240502 08:19:15.558970 22072 raft_consensus.cc:3037] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.559065 22206 raft_consensus.cc:3037] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.560763 22072 raft_consensus.cc:2454] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:15.561014 22206 raft_consensus.cc:2454] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:15.561161 22274 leader_election.cc:302] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:15.561455 22891 raft_consensus.cc:2781] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:15.561859 22891 raft_consensus.cc:686] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Becoming Leader. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Running, Role: LEADER
I20240502 08:19:15.562146 22891 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.564733 21925 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67). New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:15.588869 22891 raft_consensus.cc:482] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:15.589082 22891 raft_consensus.cc:504] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.589676 22891 leader_election.cc:288] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597), d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747)
I20240502 08:19:15.590283 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8069f943c7234829b59c930515491ad5" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031" is_pre_election: true
I20240502 08:19:15.590325 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8069f943c7234829b59c930515491ad5" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" is_pre_election: true
I20240502 08:19:15.590509 22206 raft_consensus.cc:2454] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:15.590564 22072 raft_consensus.cc:2454] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:15.590962 22274 leader_election.cc:302] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:15.591243 22891 raft_consensus.cc:2781] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:15.591329 22891 raft_consensus.cc:482] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:15.591424 22891 raft_consensus.cc:3037] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.593396 22891 raft_consensus.cc:504] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.593896 22891 leader_election.cc:288] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Requested vote from peers 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597), d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747)
I20240502 08:19:15.594399 22206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8069f943c7234829b59c930515491ad5" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031"
I20240502 08:19:15.594476 22072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8069f943c7234829b59c930515491ad5" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
I20240502 08:19:15.594686 22206 raft_consensus.cc:3037] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.594689 22072 raft_consensus.cc:3037] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:15.597543 22206 raft_consensus.cc:2454] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:15.597558 22072 raft_consensus.cc:2454] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:15.597996 22274 leader_election.cc:302] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:15.598278 22891 raft_consensus.cc:2781] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:15.598454 22891 raft_consensus.cc:686] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Becoming Leader. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Running, Role: LEADER
I20240502 08:19:15.598763 22891 consensus_queue.cc:237] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:15.601341 21925 catalog_manager.cc:5484] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67). New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:15.965648 22891 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:15.975395 22916 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:15.979367 22814 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:15.980258 22919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:15.980782 22919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:19:15.986253 22914 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:15.992063 22919 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 08:19:16.129235 22072 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:16.130246 22925 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:16.144613 22205 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:19:16.146168 22925 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:16.198168 22924 mvcc.cc:205] Tried to move back new op lower bound from 7023157068296040448 to 7023157066134478848. Current Snapshot: MvccSnapshot[applied={T|T < 7023157068296040448}]
I20240502 08:19:16.206939 22965 mvcc.cc:205] Tried to move back new op lower bound from 7023157068296040448 to 7023157066134478848. Current Snapshot: MvccSnapshot[applied={T|T < 7023157068296040448}]
I20240502 08:19:16.239233 22962 mvcc.cc:205] Tried to move back new op lower bound from 7023157068296040448 to 7023157066134478848. Current Snapshot: MvccSnapshot[applied={T|T < 7023157068296040448}]

INSERT report
    rows total: 2000
    time total: 270.6 ms
  time per row: 0.1353 ms
I20240502 08:19:16.402988 21929 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38388:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 08:19:16.403751 21929 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:19:16.412030 22183 tablet_service.cc:1467] Processing CreateTablet for tablet f2122b14ffc2475cb8a8a18ba789f68d (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.412487 22183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2122b14ffc2475cb8a8a18ba789f68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.412459 22050 tablet_service.cc:1467] Processing CreateTablet for tablet f2122b14ffc2475cb8a8a18ba789f68d (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.412880 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2122b14ffc2475cb8a8a18ba789f68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.415493 22051 tablet_service.cc:1467] Processing CreateTablet for tablet 169d28e2ca504bcfb757b049d7aa1bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.415895 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169d28e2ca504bcfb757b049d7aa1bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.415804 22185 tablet_service.cc:1467] Processing CreateTablet for tablet 169d28e2ca504bcfb757b049d7aa1bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.416167 22185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169d28e2ca504bcfb757b049d7aa1bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.419447 22316 tablet_service.cc:1467] Processing CreateTablet for tablet f2122b14ffc2475cb8a8a18ba789f68d (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.419525 22318 tablet_service.cc:1467] Processing CreateTablet for tablet 169d28e2ca504bcfb757b049d7aa1bee (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.419898 22316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2122b14ffc2475cb8a8a18ba789f68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.420399 22317 tablet_service.cc:1467] Processing CreateTablet for tablet 74f2e5440fd949b489858fd52250f668 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.420634 22184 tablet_service.cc:1467] Processing CreateTablet for tablet 74f2e5440fd949b489858fd52250f668 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.421002 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e5440fd949b489858fd52250f668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.421582 22052 tablet_service.cc:1467] Processing CreateTablet for tablet 74f2e5440fd949b489858fd52250f668 (DEFAULT_TABLE table=rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:16.421963 22052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e5440fd949b489858fd52250f668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.426054 22317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e5440fd949b489858fd52250f668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.431542 22984 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:16.432133 22318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169d28e2ca504bcfb757b049d7aa1bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:16.432327 22985 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:16.433590 22984 tablet_bootstrap.cc:654] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.435951 22985 tablet_bootstrap.cc:654] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.438470 22984 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:16.438735 22984 ts_tablet_manager.cc:1367] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.007s	user 0.006s	sys 0.000s
I20240502 08:19:16.439731 22983 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:16.439723 22984 raft_consensus.cc:348] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.439951 22984 raft_consensus.cc:374] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.439966 22985 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:16.440068 22984 raft_consensus.cc:729] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.440106 22985 ts_tablet_manager.cc:1367] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.000s
I20240502 08:19:16.440335 22984 consensus_queue.cc:260] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.440954 22984 ts_tablet_manager.cc:1398] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:16.440886 22985 raft_consensus.cc:348] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.441118 22985 raft_consensus.cc:374] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.441223 22985 raft_consensus.cc:729] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.441330 22984 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:16.441463 22985 consensus_queue.cc:260] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.441744 22983 tablet_bootstrap.cc:654] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.442247 22985 ts_tablet_manager.cc:1398] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:16.442919 22984 tablet_bootstrap.cc:654] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.445753 22985 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:16.448791 22985 tablet_bootstrap.cc:654] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.449456 22984 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:16.449648 22984 ts_tablet_manager.cc:1367] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.008s	user 0.007s	sys 0.000s
I20240502 08:19:16.449734 22983 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:16.449891 22983 ts_tablet_manager.cc:1367] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.010s	user 0.001s	sys 0.007s
I20240502 08:19:16.450443 22984 raft_consensus.cc:348] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.450769 22984 raft_consensus.cc:374] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.450872 22984 raft_consensus.cc:729] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.450763 22983 raft_consensus.cc:348] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.450975 22983 raft_consensus.cc:374] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.451079 22983 raft_consensus.cc:729] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.451155 22984 consensus_queue.cc:260] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.451328 22983 consensus_queue.cc:260] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.451750 22985 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:16.451761 22984 ts_tablet_manager.cc:1398] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:19:16.451980 22985 ts_tablet_manager.cc:1367] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 08:19:16.452263 22984 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:16.452651 22983 ts_tablet_manager.cc:1398] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:16.452771 22985 raft_consensus.cc:348] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.452965 22983 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:16.453029 22985 raft_consensus.cc:374] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.453121 22985 raft_consensus.cc:729] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.453393 22985 consensus_queue.cc:260] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.453948 22984 tablet_bootstrap.cc:654] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.454092 22985 ts_tablet_manager.cc:1398] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:19:16.455219 22983 tablet_bootstrap.cc:654] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.459393 22985 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Bootstrap starting.
I20240502 08:19:16.461674 22985 tablet_bootstrap.cc:654] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.462994 22984 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031: No bootstrap required, opened a new log
I20240502 08:19:16.463204 22984 ts_tablet_manager.cc:1367] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.011s	user 0.007s	sys 0.003s
I20240502 08:19:16.463685 22983 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:16.463851 22983 ts_tablet_manager.cc:1367] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.011s	user 0.003s	sys 0.005s
I20240502 08:19:16.463948 22985 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: No bootstrap required, opened a new log
I20240502 08:19:16.464103 22985 ts_tablet_manager.cc:1367] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:19:16.464128 22984 raft_consensus.cc:348] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.464358 22984 raft_consensus.cc:374] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.464455 22984 raft_consensus.cc:729] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.464655 22983 raft_consensus.cc:348] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.464696 22984 consensus_queue.cc:260] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.464864 22983 raft_consensus.cc:374] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.464947 22983 raft_consensus.cc:729] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.464931 22985 raft_consensus.cc:348] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.465127 22985 raft_consensus.cc:374] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.465211 22985 raft_consensus.cc:729] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.465165 22983 consensus_queue.cc:260] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.465458 22985 consensus_queue.cc:260] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.465636 22983 ts_tablet_manager.cc:1398] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:19:16.465976 22985 ts_tablet_manager.cc:1398] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:16.467109 22983 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:16.467754 22984 ts_tablet_manager.cc:1398] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.004s	user 0.000s	sys 0.002s
I20240502 08:19:16.468663 22983 tablet_bootstrap.cc:654] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:16.470609 22983 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8: No bootstrap required, opened a new log
I20240502 08:19:16.470770 22983 ts_tablet_manager.cc:1367] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.004s	user 0.001s	sys 0.002s
I20240502 08:19:16.471449 22983 raft_consensus.cc:348] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.471597 22983 raft_consensus.cc:374] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:16.471663 22983 raft_consensus.cc:729] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: FOLLOWER
I20240502 08:19:16.471850 22983 consensus_queue.cc:260] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.472242 22983 ts_tablet_manager.cc:1398] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:16.565665 22976 raft_consensus.cc:482] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:16.565920 22976 raft_consensus.cc:504] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.566618 22976 leader_election.cc:288] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:16.567288 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e5440fd949b489858fd52250f668" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031" is_pre_election: true
I20240502 08:19:16.567307 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e5440fd949b489858fd52250f668" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" is_pre_election: true
I20240502 08:19:16.567543 22070 raft_consensus.cc:2454] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:16.567564 22205 raft_consensus.cc:2454] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:16.568042 22274 leader_election.cc:302] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:16.568285 22976 raft_consensus.cc:2781] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:16.568408 22976 raft_consensus.cc:482] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:16.568501 22976 raft_consensus.cc:3037] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.570407 22976 raft_consensus.cc:504] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.570966 22976 leader_election.cc:288] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Requested vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:16.571456 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e5440fd949b489858fd52250f668" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
I20240502 08:19:16.571507 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74f2e5440fd949b489858fd52250f668" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031"
I20240502 08:19:16.571628 22070 raft_consensus.cc:3037] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.571743 22205 raft_consensus.cc:3037] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.573534 22070 raft_consensus.cc:2454] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:16.573632 22205 raft_consensus.cc:2454] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:16.573936 22274 leader_election.cc:302] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:16.574205 22976 raft_consensus.cc:2781] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:16.574385 22976 raft_consensus.cc:686] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Becoming Leader. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Running, Role: LEADER
I20240502 08:19:16.574716 22976 consensus_queue.cc:237] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.577576 21930 catalog_manager.cc:5484] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67). New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:16.654119 22907 raft_consensus.cc:482] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:16.654320 22907 raft_consensus.cc:504] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.655323 22907 leader_election.cc:288] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597), ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:16.656046 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2122b14ffc2475cb8a8a18ba789f68d" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031" is_pre_election: true
I20240502 08:19:16.656299 22205 raft_consensus.cc:2454] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 0.
I20240502 08:19:16.656260 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2122b14ffc2475cb8a8a18ba789f68d" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" is_pre_election: true
I20240502 08:19:16.656493 22338 raft_consensus.cc:2454] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 0.
I20240502 08:19:16.656749 22008 leader_election.cc:302] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, d24c3c61ac4a4eefa0e4a5d9cef079b8; no voters: 
I20240502 08:19:16.657090 22907 raft_consensus.cc:2781] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:16.657229 22907 raft_consensus.cc:482] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:16.657351 22907 raft_consensus.cc:3037] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.659394 22907 raft_consensus.cc:504] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.659929 22907 leader_election.cc:288] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 election: Requested vote from peers 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597), ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:16.660362 22338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2122b14ffc2475cb8a8a18ba789f68d" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3"
I20240502 08:19:16.660413 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2122b14ffc2475cb8a8a18ba789f68d" candidate_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031"
I20240502 08:19:16.660573 22338 raft_consensus.cc:3037] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.660584 22205 raft_consensus.cc:3037] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.662497 22205 raft_consensus.cc:2454] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 1.
I20240502 08:19:16.662665 22338 raft_consensus.cc:2454] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24c3c61ac4a4eefa0e4a5d9cef079b8 in term 1.
I20240502 08:19:16.663094 22008 leader_election.cc:302] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, d24c3c61ac4a4eefa0e4a5d9cef079b8; no voters: 
I20240502 08:19:16.663441 22907 raft_consensus.cc:2781] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:16.664000 22907 raft_consensus.cc:686] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Becoming Leader. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Running, Role: LEADER
I20240502 08:19:16.664319 22907 consensus_queue.cc:237] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.667372 21930 catalog_manager.cc:5484] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:16.737287 22976 raft_consensus.cc:482] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:16.737473 22976 raft_consensus.cc:504] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.738060 22976 leader_election.cc:288] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:16.738735 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169d28e2ca504bcfb757b049d7aa1bee" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" is_pre_election: true
I20240502 08:19:16.738735 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169d28e2ca504bcfb757b049d7aa1bee" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031" is_pre_election: true
I20240502 08:19:16.738984 22205 raft_consensus.cc:2454] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:16.739012 22070 raft_consensus.cc:2454] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 0.
I20240502 08:19:16.739421 22274 leader_election.cc:302] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef0b298bd6642ea900f3e9843bc3031, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:16.739697 22976 raft_consensus.cc:2781] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:16.739799 22976 raft_consensus.cc:482] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:16.739910 22976 raft_consensus.cc:3037] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.741837 22976 raft_consensus.cc:504] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.742363 22976 leader_election.cc:288] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Requested vote from peers d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747), 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:16.742782 22070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169d28e2ca504bcfb757b049d7aa1bee" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8"
I20240502 08:19:16.742875 22205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169d28e2ca504bcfb757b049d7aa1bee" candidate_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef0b298bd6642ea900f3e9843bc3031"
I20240502 08:19:16.742985 22070 raft_consensus.cc:3037] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.743099 22205 raft_consensus.cc:3037] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:16.745168 22070 raft_consensus.cc:2454] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:16.745481 22205 raft_consensus.cc:2454] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecc8a9127fcc4e3a8befd1e0ddfebeb3 in term 1.
I20240502 08:19:16.745569 22274 leader_election.cc:302] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24c3c61ac4a4eefa0e4a5d9cef079b8, ecc8a9127fcc4e3a8befd1e0ddfebeb3; no voters: 
I20240502 08:19:16.745834 22976 raft_consensus.cc:2781] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:16.746003 22976 raft_consensus.cc:686] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Becoming Leader. State: Replica: ecc8a9127fcc4e3a8befd1e0ddfebeb3, State: Running, Role: LEADER
I20240502 08:19:16.746277 22976 consensus_queue.cc:237] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } }
I20240502 08:19:16.748937 21930 catalog_manager.cc:5484] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67). New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:16.979606 22976 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:16.985517 22976 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:17.078330 22907 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:17.083181 22907 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:17.154078 22999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:17.154451 22999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:17.162685 22999 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:19:17.214084 22891 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:17.220813 22976 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 231.122 ms
  time per row: 0.115561 ms
I20240502 08:19:17.539458  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.70:46149
--local_ip_for_outbound_sockets=127.6.151.70
--tserver_master_addrs=127.6.151.126:44205
--webserver_port=33159
--webserver_interface=127.6.151.70
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:19:17.892355 23054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:17.892659 23054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:17.892793 23054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:17.900651 23054 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:17.900806 23054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:17.900972 23054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.70
I20240502 08:19:17.910310 23054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.70:46149
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.151.70
--webserver_port=33159
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.70
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:17.911712 23054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:17.914062 23054 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:17.928171 23062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:17.928884 23059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:17.929126 23060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:17.930131 23054 server_base.cc:1034] running on GCE node
I20240502 08:19:17.930824 23054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:17.931897 23054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:17.933130 23054 hybrid_clock.cc:648] HybridClock initialized: now 1714637957933077 us; error 45 us; skew 500 ppm
I20240502 08:19:17.933478 23054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:17.936105 23054 webserver.cc:460] Webserver started at http://127.6.151.70:33159/ using document root <none> and password file <none>
I20240502 08:19:17.936890 23054 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:17.937045 23054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:17.942626 23054 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.002s	sys 0.001s
I20240502 08:19:17.945765 23068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:17.947261 23054 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:19:17.947398 23054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fdea103695a94a099acd05c42bcccdca"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:17.948275 23054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:17.977241 23054 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:17.978098 23054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:17.978350 23054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:17.979339 23054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:17.980866 23054 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:17.980942 23054 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:17.981040 23054 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:17.981084 23054 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:18.013182 23054 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.70:46149
I20240502 08:19:18.013244 23180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.70:46149 every 8 connection(s)
I20240502 08:19:18.014708 23054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 08:19:18.020463  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 23054
I20240502 08:19:18.021250  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:41139
--local_ip_for_outbound_sockets=127.6.151.69
--tserver_master_addrs=127.6.151.126:44205
--webserver_port=45493
--webserver_interface=127.6.151.69
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:18.038276 23181 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:18.038699 23181 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:18.039716 23181 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:18.041513 21923 ts_manager.cc:194] Re-registered known tserver with Master: fdea103695a94a099acd05c42bcccdca (127.6.151.70:46149)
I20240502 08:19:18.042436 21923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.70:48839
W20240502 08:19:18.372887 23185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:18.373181 23185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:18.373318 23185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:18.380928 23185 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:18.381059 23185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:18.381224 23185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:19:18.390857 23185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:41139
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=45493
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:18.392194 23185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:18.393914 23185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:18.407137 23191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:18.407280 23193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:18.407699 23190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:18.418718 23185 server_base.cc:1034] running on GCE node
I20240502 08:19:18.419389 23185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:18.420428 23185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:18.421638 23185 hybrid_clock.cc:648] HybridClock initialized: now 1714637958421593 us; error 42 us; skew 500 ppm
I20240502 08:19:18.421957 23185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:18.424566 23185 webserver.cc:460] Webserver started at http://127.6.151.69:45493/ using document root <none> and password file <none>
I20240502 08:19:18.425447 23185 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:18.425609 23185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:18.431227 23185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s	user 0.001s	sys 0.003s
I20240502 08:19:18.433955 23199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:18.435376 23185 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:19:18.435537 23185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "26d5ad8a5f594649b73ab2b11dc3d328"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:18.436352 23185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:18.459774 23185 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:18.460633 23185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:18.460911 23185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:18.461880 23185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:18.463479 23185 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:18.463558 23185 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:18.463685 23185 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:18.463740 23185 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:18.498422 23185 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:41139
I20240502 08:19:18.498474 23311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:41139 every 8 connection(s)
I20240502 08:19:18.499994 23185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:19:18.508347  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 23185
I20240502 08:19:18.509003  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:34181
--local_ip_for_outbound_sockets=127.6.151.68
--tserver_master_addrs=127.6.151.126:44205
--webserver_port=38779
--webserver_interface=127.6.151.68
--builtin_ntp_servers=127.6.151.84:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:18.513733 23312 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:18.514143 23312 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:18.515240 23312 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:18.516918 21923 ts_manager.cc:194] Re-registered known tserver with Master: 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69:41139)
I20240502 08:19:18.518061 21923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:57753
W20240502 08:19:18.866516 23315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:18.866919 23315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:18.867067 23315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:18.875386 23315 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:18.875545 23315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:18.875705 23315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:19:18.885227 23315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:34181
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=38779
--tserver_master_addrs=127.6.151.126:44205
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:18.886583 23315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:18.888427 23315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:18.900803 23321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:18.901062 23324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:18.901700 23322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:18.902046 23315 server_base.cc:1034] running on GCE node
I20240502 08:19:18.902686 23315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:18.903573 23315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:18.904773 23315 hybrid_clock.cc:648] HybridClock initialized: now 1714637958904724 us; error 45 us; skew 500 ppm
I20240502 08:19:18.905110 23315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:18.907339 23315 webserver.cc:460] Webserver started at http://127.6.151.68:38779/ using document root <none> and password file <none>
I20240502 08:19:18.908063 23315 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:18.908202 23315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:18.913260 23315 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.004s
I20240502 08:19:18.915925 23330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:18.917207 23315 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:19:18.917352 23315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a"
format_stamp: "Formatted at 2024-05-02 08:19:08 on dist-test-slave-x3qn"
I20240502 08:19:18.918164 23315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:18.949846 23315 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:18.950927 23315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:18.951263 23315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:18.952256 23315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:18.953853 23315 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:18.953929 23315 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:18.954056 23315 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:18.954106 23315 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:18.986596 23315 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:34181
I20240502 08:19:18.986675 23442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:34181 every 8 connection(s)
I20240502 08:19:18.988221 23315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:19:18.994016  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 23315
I20240502 08:19:19.001641 23443 heartbeater.cc:347] Connected to a master server at 127.6.151.126:44205
I20240502 08:19:19.002015 23443 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:19.002959 23443 heartbeater.cc:511] Master 127.6.151.126:44205 requested a full tablet report, sending...
I20240502 08:19:19.004513 21923 ts_manager.cc:194] Re-registered known tserver with Master: b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181)
I20240502 08:19:19.005308 21923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:37329
I20240502 08:19:19.045706 23181 heartbeater.cc:503] Master 127.6.151.126:44205 was elected leader, sending a full tablet report...
I20240502 08:19:19.502899 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:19.503154 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.511643 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.521229 23312 heartbeater.cc:503] Master 127.6.151.126:44205 was elected leader, sending a full tablet report...
I20240502 08:19:19.533198 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:19.555402 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:19.557858 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.734395 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:19.735837 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.743618 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:19.751473 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:19.764750 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.787762 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.964589 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.970341 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:19.973886 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:19.993486 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:19.996502 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:20.007859 23443 heartbeater.cc:503] Master 127.6.151.126:44205 was elected leader, sending a full tablet report...
I20240502 08:19:20.010591 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:20.178489 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:20.188889 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:20.193236 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:20.204381 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:20.210984 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:20.213001 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:20.296221 22338 consensus_queue.cc:237] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:20.299754 22072 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.299785 22206 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.300405 23593 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.300822 23594 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.304193 23593 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.304805 22072 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
W20240502 08:19:20.308200 22272 consensus_peers.cc:485] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69:41139): Couldn't send request to peer 26d5ad8a5f594649b73ab2b11dc3d328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 169d28e2ca504bcfb757b049d7aa1bee. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.310905 21923 catalog_manager.cc:5484] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.313469 22205 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.331722 22338 consensus_queue.cc:237] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:20.334388 22072 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.334404 22206 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.335045 23598 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.335422 23593 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.338498 23593 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.339480 22072 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.339547 22206 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
W20240502 08:19:20.344709 22274 consensus_peers.cc:485] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181): Couldn't send request to peer b6ca0fbc79ad4b1780eebe8bc88c733a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8069f943c7234829b59c930515491ad5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.344949 21927 catalog_manager.cc:5484] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:20.364617 22206 consensus_queue.cc:237] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:20.367801 22072 raft_consensus.cc:1261] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.368351 22338 raft_consensus.cc:1261] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.368471 23618 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.369050 22971 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.372598 22971 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:20.374454 22070 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:20.375028 22338 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
W20240502 08:19:20.379926 22139 consensus_peers.cc:485] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 -> Peer fdea103695a94a099acd05c42bcccdca (127.6.151.70:46149): Couldn't send request to peer fdea103695a94a099acd05c42bcccdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd3b3b39b7b54ef3b7c3b5e7265b76de. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.380266 21930 catalog_manager.cc:5484] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:20.393848 22070 consensus_queue.cc:237] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:20.396981 22338 raft_consensus.cc:1261] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.397487 22206 raft_consensus.cc:1261] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.397621 22996 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.398087 23628 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.401679 22996 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.402405 22206 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
W20240502 08:19:20.403517 22006 consensus_peers.cc:485] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 -> Peer 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69:41139): Couldn't send request to peer 26d5ad8a5f594649b73ab2b11dc3d328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5507342ac56b42878ce02a2dcf149bc0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.405308 22338 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.406895 21927 catalog_manager.cc:5484] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.422510 22338 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
W20240502 08:19:20.425029 22274 consensus_peers.cc:485] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181): Couldn't send request to peer b6ca0fbc79ad4b1780eebe8bc88c733a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82db7fca82b5492faa762066294de080. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.425042 22206 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.425086 22070 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.425598 23593 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.425902 23594 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.428611 23598 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.429342 22206 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.429409 22070 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.433787 21923 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.448747 22070 consensus_queue.cc:237] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:20.451262 22206 raft_consensus.cc:1261] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.451833 23632 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.451875 22338 raft_consensus.cc:1261] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.452497 23632 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.454931 23633 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:20.455530 22206 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:20.457041 22338 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
W20240502 08:19:20.457660 22006 consensus_peers.cc:485] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 -> Peer fdea103695a94a099acd05c42bcccdca (127.6.151.70:46149): Couldn't send request to peer fdea103695a94a099acd05c42bcccdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2122b14ffc2475cb8a8a18ba789f68d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.460460 21927 catalog_manager.cc:5484] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index -1 to 4, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:20.473852 22338 consensus_queue.cc:237] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:20.476330 22070 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.476539 22206 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:19:20.476655 22272 consensus_peers.cc:485] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69:41139): Couldn't send request to peer 26d5ad8a5f594649b73ab2b11dc3d328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74f2e5440fd949b489858fd52250f668. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.476956 23594 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.477249 23316 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.480101 23594 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.480844 22206 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.483933 22070 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.486024 21930 catalog_manager.cc:5484] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index -1 to 4, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:20.500380 22206 consensus_queue.cc:237] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:20.502955 22338 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.503393 22070 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:20.503854 23618 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.505084 23618 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:20.508065 23618 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.508520 22338 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:20.510998 22070 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
W20240502 08:19:20.511662 22141 consensus_peers.cc:485] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 -> Peer b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181): Couldn't send request to peer b6ca0fbc79ad4b1780eebe8bc88c733a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cd539305b35444596c668a83ace2fbd. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:20.513830 21930 catalog_manager.cc:5484] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index -1 to 4, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:20.707511 23707 ts_tablet_manager.cc:902] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:20.709280 23707 tablet_copy_client.cc:308] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:20.715935 22358 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 169d28e2ca504bcfb757b049d7aa1bee from peer 26d5ad8a5f594649b73ab2b11dc3d328 ({username='slave'} at 127.6.151.69:35553)
I20240502 08:19:20.716187 22358 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 169d28e2ca504bcfb757b049d7aa1bee from peer 26d5ad8a5f594649b73ab2b11dc3d328 at {username='slave'} at 127.6.151.69:35553: session id = 26d5ad8a5f594649b73ab2b11dc3d328-169d28e2ca504bcfb757b049d7aa1bee
I20240502 08:19:20.718873 22358 tablet_copy_source_session.cc:205] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:20.723554 23707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169d28e2ca504bcfb757b049d7aa1bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:20.733785 23707 tablet_copy_client.cc:783] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:20.734140 23707 tablet_copy_client.cc:647] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:20.736599 23707 tablet_copy_client.cc:523] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:20.740013 23707 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap starting.
I20240502 08:19:20.750072 23707 log.cc:824] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:20.781167 23707 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:20.781739 23707 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap complete.
I20240502 08:19:20.782173 23707 ts_tablet_manager.cc:1367] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent bootstrapping tablet: real 0.042s	user 0.037s	sys 0.003s
I20240502 08:19:20.786594 23707 raft_consensus.cc:348] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:20.787074 23707 raft_consensus.cc:729] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 26d5ad8a5f594649b73ab2b11dc3d328, State: Initialized, Role: LEARNER
I20240502 08:19:20.787657 23707 consensus_queue.cc:260] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:20.789597 23707 ts_tablet_manager.cc:1398] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.004s
I20240502 08:19:20.790832 22358 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-169d28e2ca504bcfb757b049d7aa1bee received from {username='slave'} at 127.6.151.69:35553
I20240502 08:19:20.791024 22358 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-169d28e2ca504bcfb757b049d7aa1bee on tablet 169d28e2ca504bcfb757b049d7aa1bee with peer 26d5ad8a5f594649b73ab2b11dc3d328
I20240502 08:19:20.832777 23740 ts_tablet_manager.cc:902] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:20.834795 23740 tablet_copy_client.cc:308] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:20.854444 22358 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 8069f943c7234829b59c930515491ad5 from peer b6ca0fbc79ad4b1780eebe8bc88c733a ({username='slave'} at 127.6.151.68:45023)
I20240502 08:19:20.854607 22358 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 8069f943c7234829b59c930515491ad5 from peer b6ca0fbc79ad4b1780eebe8bc88c733a at {username='slave'} at 127.6.151.68:45023: session id = b6ca0fbc79ad4b1780eebe8bc88c733a-8069f943c7234829b59c930515491ad5
I20240502 08:19:20.857128 22358 tablet_copy_source_session.cc:205] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:20.862414 23740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8069f943c7234829b59c930515491ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:20.875387 23740 tablet_copy_client.cc:783] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:20.875797 23740 tablet_copy_client.cc:647] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:20.878187 23740 tablet_copy_client.cc:523] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:20.881321 23740 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap starting.
I20240502 08:19:20.891036 23740 log.cc:824] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:20.922322 23740 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:20.922959 23740 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap complete.
I20240502 08:19:20.923462 23740 ts_tablet_manager.cc:1367] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent bootstrapping tablet: real 0.042s	user 0.043s	sys 0.001s
I20240502 08:19:20.925153 23707 ts_tablet_manager.cc:902] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:20.926560 23707 tablet_copy_client.cc:308] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:20.927417 22358 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 74f2e5440fd949b489858fd52250f668 from peer 26d5ad8a5f594649b73ab2b11dc3d328 ({username='slave'} at 127.6.151.69:35553)
I20240502 08:19:20.927582 22358 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 74f2e5440fd949b489858fd52250f668 from peer 26d5ad8a5f594649b73ab2b11dc3d328 at {username='slave'} at 127.6.151.69:35553: session id = 26d5ad8a5f594649b73ab2b11dc3d328-74f2e5440fd949b489858fd52250f668
I20240502 08:19:20.928948 23740 raft_consensus.cc:348] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:20.929360 23740 raft_consensus.cc:729] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6ca0fbc79ad4b1780eebe8bc88c733a, State: Initialized, Role: LEARNER
I20240502 08:19:20.929805 22358 tablet_copy_source_session.cc:205] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:20.929853 23740 consensus_queue.cc:260] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:20.931221 23707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e5440fd949b489858fd52250f668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:20.931679 23740 ts_tablet_manager.cc:1398] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20240502 08:19:20.932485 22358 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-8069f943c7234829b59c930515491ad5 received from {username='slave'} at 127.6.151.68:45023
I20240502 08:19:20.932618 22358 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-8069f943c7234829b59c930515491ad5 on tablet 8069f943c7234829b59c930515491ad5 with peer b6ca0fbc79ad4b1780eebe8bc88c733a
I20240502 08:19:20.937399 23707 tablet_copy_client.cc:783] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:20.937937 23707 tablet_copy_client.cc:647] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:20.939433 23707 tablet_copy_client.cc:523] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:20.942054 23707 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap starting.
I20240502 08:19:20.953670 23768 ts_tablet_manager.cc:902] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: Initiating tablet copy from peer d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747)
I20240502 08:19:20.955286 23768 tablet_copy_client.cc:308] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:45747
I20240502 08:19:20.961438 22092 tablet_copy_service.cc:140] P d24c3c61ac4a4eefa0e4a5d9cef079b8: Received BeginTabletCopySession request for tablet 5507342ac56b42878ce02a2dcf149bc0 from peer 26d5ad8a5f594649b73ab2b11dc3d328 ({username='slave'} at 127.6.151.69:33411)
I20240502 08:19:20.961648 22092 tablet_copy_service.cc:161] P d24c3c61ac4a4eefa0e4a5d9cef079b8: Beginning new tablet copy session on tablet 5507342ac56b42878ce02a2dcf149bc0 from peer 26d5ad8a5f594649b73ab2b11dc3d328 at {username='slave'} at 127.6.151.69:33411: session id = 26d5ad8a5f594649b73ab2b11dc3d328-5507342ac56b42878ce02a2dcf149bc0
I20240502 08:19:20.964442 22092 tablet_copy_source_session.cc:205] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:20.966153 23768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5507342ac56b42878ce02a2dcf149bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:20.973006 23768 tablet_copy_client.cc:783] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:20.973273 23768 tablet_copy_client.cc:647] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:20.975615 23768 tablet_copy_client.cc:523] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:20.978011 23740 ts_tablet_manager.cc:902] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:20.979276 23768 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap starting.
I20240502 08:19:20.981281 23707 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:20.981765 23707 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap complete.
I20240502 08:19:20.982079 23707 ts_tablet_manager.cc:1367] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent bootstrapping tablet: real 0.040s	user 0.035s	sys 0.003s
I20240502 08:19:20.983175 23707 raft_consensus.cc:348] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:20.983461 23707 raft_consensus.cc:729] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 26d5ad8a5f594649b73ab2b11dc3d328, State: Initialized, Role: LEARNER
I20240502 08:19:20.983655 23707 consensus_queue.cc:260] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:20.985005 23707 ts_tablet_manager.cc:1398] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
I20240502 08:19:20.985704 22358 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-74f2e5440fd949b489858fd52250f668 received from {username='slave'} at 127.6.151.69:35553
I20240502 08:19:20.985867 22358 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-74f2e5440fd949b489858fd52250f668 on tablet 74f2e5440fd949b489858fd52250f668 with peer 26d5ad8a5f594649b73ab2b11dc3d328
I20240502 08:19:20.995296 23740 tablet_copy_client.cc:308] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:20.996371 22358 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 82db7fca82b5492faa762066294de080 from peer b6ca0fbc79ad4b1780eebe8bc88c733a ({username='slave'} at 127.6.151.68:45023)
I20240502 08:19:20.996568 22358 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 82db7fca82b5492faa762066294de080 from peer b6ca0fbc79ad4b1780eebe8bc88c733a at {username='slave'} at 127.6.151.68:45023: session id = b6ca0fbc79ad4b1780eebe8bc88c733a-82db7fca82b5492faa762066294de080
I20240502 08:19:20.999673 22358 tablet_copy_source_session.cc:205] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:21.002600 23740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82db7fca82b5492faa762066294de080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:21.007575 23778 ts_tablet_manager.cc:902] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: Initiating tablet copy from peer 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:21.009647 23740 tablet_copy_client.cc:783] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:21.009912 23740 tablet_copy_client.cc:647] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:21.011755 23740 tablet_copy_client.cc:523] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:21.022861 23740 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap starting.
I20240502 08:19:21.025918 23781 ts_tablet_manager.cc:902] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: Initiating tablet copy from peer 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:21.047397 23783 ts_tablet_manager.cc:902] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: Initiating tablet copy from peer d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747)
I20240502 08:19:21.063560 23783 tablet_copy_client.cc:308] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:45747
I20240502 08:19:21.090481 22092 tablet_copy_service.cc:140] P d24c3c61ac4a4eefa0e4a5d9cef079b8: Received BeginTabletCopySession request for tablet f2122b14ffc2475cb8a8a18ba789f68d from peer fdea103695a94a099acd05c42bcccdca ({username='slave'} at 127.6.151.70:42717)
I20240502 08:19:21.090750 22092 tablet_copy_service.cc:161] P d24c3c61ac4a4eefa0e4a5d9cef079b8: Beginning new tablet copy session on tablet f2122b14ffc2475cb8a8a18ba789f68d from peer fdea103695a94a099acd05c42bcccdca at {username='slave'} at 127.6.151.70:42717: session id = fdea103695a94a099acd05c42bcccdca-f2122b14ffc2475cb8a8a18ba789f68d
I20240502 08:19:21.094233 22092 tablet_copy_source_session.cc:205] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:21.097986 23781 tablet_copy_client.cc:308] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:42597
I20240502 08:19:21.112067 23783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2122b14ffc2475cb8a8a18ba789f68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:21.112188 22226 tablet_copy_service.cc:140] P 8ef0b298bd6642ea900f3e9843bc3031: Received BeginTabletCopySession request for tablet 4cd539305b35444596c668a83ace2fbd from peer b6ca0fbc79ad4b1780eebe8bc88c733a ({username='slave'} at 127.6.151.68:59835)
I20240502 08:19:21.112473 22226 tablet_copy_service.cc:161] P 8ef0b298bd6642ea900f3e9843bc3031: Beginning new tablet copy session on tablet 4cd539305b35444596c668a83ace2fbd from peer b6ca0fbc79ad4b1780eebe8bc88c733a at {username='slave'} at 127.6.151.68:59835: session id = b6ca0fbc79ad4b1780eebe8bc88c733a-4cd539305b35444596c668a83ace2fbd
I20240502 08:19:21.117686 22226 tablet_copy_source_session.cc:205] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:21.120478 23781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd539305b35444596c668a83ace2fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:21.123093 23778 tablet_copy_client.cc:308] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:42597
I20240502 08:19:21.125818 23768 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:21.129981 23768 tablet_bootstrap.cc:492] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap complete.
I20240502 08:19:21.130306 23768 ts_tablet_manager.cc:1367] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent bootstrapping tablet: real 0.151s	user 0.049s	sys 0.010s
I20240502 08:19:21.130820 23781 tablet_copy_client.cc:783] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:21.131095 23781 tablet_copy_client.cc:647] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:21.131201 23768 raft_consensus.cc:348] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:21.131471 23768 raft_consensus.cc:729] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 26d5ad8a5f594649b73ab2b11dc3d328, State: Initialized, Role: LEARNER
I20240502 08:19:21.131682 23768 consensus_queue.cc:260] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:21.134408 22226 tablet_copy_service.cc:140] P 8ef0b298bd6642ea900f3e9843bc3031: Received BeginTabletCopySession request for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de from peer fdea103695a94a099acd05c42bcccdca ({username='slave'} at 127.6.151.70:47563)
I20240502 08:19:21.134673 22226 tablet_copy_service.cc:161] P 8ef0b298bd6642ea900f3e9843bc3031: Beginning new tablet copy session on tablet cd3b3b39b7b54ef3b7c3b5e7265b76de from peer fdea103695a94a099acd05c42bcccdca at {username='slave'} at 127.6.151.70:47563: session id = fdea103695a94a099acd05c42bcccdca-cd3b3b39b7b54ef3b7c3b5e7265b76de
I20240502 08:19:21.137364 23781 tablet_copy_client.cc:523] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:21.138787 23768 ts_tablet_manager.cc:1398] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.000s
I20240502 08:19:21.139281 22226 tablet_copy_source_session.cc:205] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:21.141170 23781 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap starting.
I20240502 08:19:21.142244 22092 tablet_copy_service.cc:342] P d24c3c61ac4a4eefa0e4a5d9cef079b8: Request end of tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-5507342ac56b42878ce02a2dcf149bc0 received from {username='slave'} at 127.6.151.69:33411
I20240502 08:19:21.142441 22092 tablet_copy_service.cc:433] P d24c3c61ac4a4eefa0e4a5d9cef079b8: ending tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-5507342ac56b42878ce02a2dcf149bc0 on tablet 5507342ac56b42878ce02a2dcf149bc0 with peer 26d5ad8a5f594649b73ab2b11dc3d328
I20240502 08:19:21.144277 23778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:21.149065 23740 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:21.149583 23740 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap complete.
I20240502 08:19:21.149914 23740 ts_tablet_manager.cc:1367] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent bootstrapping tablet: real 0.127s	user 0.074s	sys 0.000s
I20240502 08:19:21.151014 23740 raft_consensus.cc:348] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:21.151304 23740 raft_consensus.cc:729] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6ca0fbc79ad4b1780eebe8bc88c733a, State: Initialized, Role: LEARNER
I20240502 08:19:21.151527 23740 consensus_queue.cc:260] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:21.155242 23740 ts_tablet_manager.cc:1398] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20240502 08:19:21.157189 23783 tablet_copy_client.cc:783] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:21.157657 23783 tablet_copy_client.cc:647] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:21.159035 23778 tablet_copy_client.cc:783] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:21.159324 23778 tablet_copy_client.cc:647] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:21.160749 23783 tablet_copy_client.cc:523] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:21.161084 22358 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-82db7fca82b5492faa762066294de080 received from {username='slave'} at 127.6.151.68:45023
I20240502 08:19:21.161224 22358 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-82db7fca82b5492faa762066294de080 on tablet 82db7fca82b5492faa762066294de080 with peer b6ca0fbc79ad4b1780eebe8bc88c733a
I20240502 08:19:21.162063 23778 tablet_copy_client.cc:523] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:21.167372 23778 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: Bootstrap starting.
I20240502 08:19:21.167425 23783 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: Bootstrap starting.
I20240502 08:19:21.182855 23783 log.cc:824] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:21.207825 23781 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:21.208388 23781 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap complete.
I20240502 08:19:21.208681 23781 ts_tablet_manager.cc:1367] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent bootstrapping tablet: real 0.068s	user 0.054s	sys 0.008s
I20240502 08:19:21.211402 23781 raft_consensus.cc:348] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:21.211692 23781 raft_consensus.cc:729] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6ca0fbc79ad4b1780eebe8bc88c733a, State: Initialized, Role: LEARNER
I20240502 08:19:21.211913 23781 consensus_queue.cc:260] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:21.213289 23781 ts_tablet_manager.cc:1398] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:19:21.214174 22226 tablet_copy_service.cc:342] P 8ef0b298bd6642ea900f3e9843bc3031: Request end of tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-4cd539305b35444596c668a83ace2fbd received from {username='slave'} at 127.6.151.68:59835
I20240502 08:19:21.214358 22226 tablet_copy_service.cc:433] P 8ef0b298bd6642ea900f3e9843bc3031: ending tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-4cd539305b35444596c668a83ace2fbd on tablet 4cd539305b35444596c668a83ace2fbd with peer b6ca0fbc79ad4b1780eebe8bc88c733a
I20240502 08:19:21.226747 23267 raft_consensus.cc:1203] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.259742 23778 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:21.260394 23778 tablet_bootstrap.cc:492] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: Bootstrap complete.
I20240502 08:19:21.260972 23778 ts_tablet_manager.cc:1367] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: Time spent bootstrapping tablet: real 0.094s	user 0.062s	sys 0.008s
I20240502 08:19:21.261641 23783 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:21.262123 23783 tablet_bootstrap.cc:492] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: Bootstrap complete.
I20240502 08:19:21.262425 23783 ts_tablet_manager.cc:1367] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: Time spent bootstrapping tablet: real 0.095s	user 0.063s	sys 0.009s
I20240502 08:19:21.267817 23783 raft_consensus.cc:348] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:21.267817 23778 raft_consensus.cc:348] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:21.268472 23778 raft_consensus.cc:729] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdea103695a94a099acd05c42bcccdca, State: Initialized, Role: LEARNER
I20240502 08:19:21.268471 23783 raft_consensus.cc:729] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdea103695a94a099acd05c42bcccdca, State: Initialized, Role: LEARNER
I20240502 08:19:21.269146 23778 consensus_queue.cc:260] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:21.269183 23783 consensus_queue.cc:260] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:21.271807 23783 ts_tablet_manager.cc:1398] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.001s
I20240502 08:19:21.272282 23778 ts_tablet_manager.cc:1398] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.001s
I20240502 08:19:21.272940 22226 tablet_copy_service.cc:342] P 8ef0b298bd6642ea900f3e9843bc3031: Request end of tablet copy session fdea103695a94a099acd05c42bcccdca-cd3b3b39b7b54ef3b7c3b5e7265b76de received from {username='slave'} at 127.6.151.70:47563
I20240502 08:19:21.272940 22092 tablet_copy_service.cc:342] P d24c3c61ac4a4eefa0e4a5d9cef079b8: Request end of tablet copy session fdea103695a94a099acd05c42bcccdca-f2122b14ffc2475cb8a8a18ba789f68d received from {username='slave'} at 127.6.151.70:42717
I20240502 08:19:21.273063 22226 tablet_copy_service.cc:433] P 8ef0b298bd6642ea900f3e9843bc3031: ending tablet copy session fdea103695a94a099acd05c42bcccdca-cd3b3b39b7b54ef3b7c3b5e7265b76de on tablet cd3b3b39b7b54ef3b7c3b5e7265b76de with peer fdea103695a94a099acd05c42bcccdca
I20240502 08:19:21.273063 22092 tablet_copy_service.cc:433] P d24c3c61ac4a4eefa0e4a5d9cef079b8: ending tablet copy session fdea103695a94a099acd05c42bcccdca-f2122b14ffc2475cb8a8a18ba789f68d on tablet f2122b14ffc2475cb8a8a18ba789f68d with peer fdea103695a94a099acd05c42bcccdca
I20240502 08:19:21.342944 23267 raft_consensus.cc:1203] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.412650 23398 raft_consensus.cc:1203] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.451999 23398 raft_consensus.cc:1203] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.468523 23136 raft_consensus.cc:1203] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.484225 23136 raft_consensus.cc:1203] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.517925 23267 raft_consensus.cc:1203] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.554469 23398 raft_consensus.cc:1203] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:21.654152 23809 raft_consensus.cc:1050] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 to VOTER
I20240502 08:19:21.655006 23809 consensus_queue.cc:237] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:21.657276 22070 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.657379 22206 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.657438 23267 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.658102 23809 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.658432 23812 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.658776 23811 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.663158 23809 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.664160 23267 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.667872 22070 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.669292 21923 catalog_manager.cc:5484] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:21.673977 22206 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.683648 22338 consensus_queue.cc:237] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:21.685344 23267 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.685740 22070 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.685930 23809 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.686475 23809 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.688221 23809 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.688841 23267 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.689404 22070 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.691974 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 169d28e2ca504bcfb757b049d7aa1bee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:21.694048 21926 catalog_manager.cc:5484] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 reported cstate change: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.697896 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 169d28e2ca504bcfb757b049d7aa1bee with delete_type TABLET_DATA_TOMBSTONED (TS 8ef0b298bd6642ea900f3e9843bc3031 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39008
I20240502 08:19:21.698841 23895 tablet_replica.cc:330] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: stopping tablet replica
I20240502 08:19:21.699265 23895 raft_consensus.cc:2229] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:21.699554 23895 raft_consensus.cc:2258] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:21.700896 23895 ts_tablet_manager.cc:1875] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:21.705427 23895 ts_tablet_manager.cc:1888] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:21.705559 23895 log.cc:1196] T 169d28e2ca504bcfb757b049d7aa1bee P 8ef0b298bd6642ea900f3e9843bc3031: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/169d28e2ca504bcfb757b049d7aa1bee
I20240502 08:19:21.706305 21918 catalog_manager.cc:4830] TS 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597): tablet 169d28e2ca504bcfb757b049d7aa1bee (table rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]) successfully deleted
I20240502 08:19:21.799716 23809 raft_consensus.cc:1050] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a to VOTER
I20240502 08:19:21.800552 23809 consensus_queue.cc:237] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:21.804026 22070 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.804075 22206 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.804220 23398 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.804522 23811 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.804792 23812 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.805653 23809 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.809721 23812 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.811223 22070 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.811349 23398 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.816835 21926 catalog_manager.cc:5484] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.818684 22206 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.824064 22338 consensus_queue.cc:237] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:21.825796 23398 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.826413 22206 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.826423 23316 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.827086 23316 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.829325 23812 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.830554 22206 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.830981 23398 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.833328 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8069f943c7234829b59c930515491ad5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:21.834620 21930 catalog_manager.cc:5484] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:21.839788 22050 tablet_service.cc:1514] Processing DeleteTablet for tablet 8069f943c7234829b59c930515491ad5 with delete_type TABLET_DATA_TOMBSTONED (TS d24c3c61ac4a4eefa0e4a5d9cef079b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43852
I20240502 08:19:21.843508 23911 tablet_replica.cc:330] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: stopping tablet replica
I20240502 08:19:21.843886 23911 raft_consensus.cc:2229] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:21.844236 23911 raft_consensus.cc:2258] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:21.845615 23911 ts_tablet_manager.cc:1875] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:21.848263 23845 raft_consensus.cc:1050] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031: attempting to promote NON_VOTER fdea103695a94a099acd05c42bcccdca to VOTER
I20240502 08:19:21.849328 23845 consensus_queue.cc:237] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } }
I20240502 08:19:21.852084 22338 raft_consensus.cc:1261] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.852468 23911 ts_tablet_manager.cc:1888] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:21.852586 23136 raft_consensus.cc:1261] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.852615 23911 log.cc:1196] T 8069f943c7234829b59c930515491ad5 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/8069f943c7234829b59c930515491ad5
I20240502 08:19:21.853299 23846 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.853591 21918 catalog_manager.cc:4830] TS d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747): tablet 8069f943c7234829b59c930515491ad5 (table rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]) successfully deleted
I20240502 08:19:21.853690 23617 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.854466 22070 raft_consensus.cc:1261] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.857275 23844 raft_consensus.cc:1050] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8: attempting to promote NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 to VOTER
I20240502 08:19:21.857498 23846 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.858376 23844 consensus_queue.cc:237] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:21.859186 23846 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.863421 22338 raft_consensus.cc:1261] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.863467 22206 raft_consensus.cc:1261] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.864107 23843 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.863847 23135 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.867749 23267 raft_consensus.cc:1261] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.868216 22996 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.869632 22996 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.879904 21923 catalog_manager.cc:5484] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca reported cstate change: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.880757 23843 raft_consensus.cc:1050] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8: attempting to promote NON_VOTER fdea103695a94a099acd05c42bcccdca to VOTER
I20240502 08:19:21.881810 23843 consensus_queue.cc:237] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } }
I20240502 08:19:21.883581 22337 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.890501 22996 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.893464 22071 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.899507 22338 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.906535 23135 raft_consensus.cc:1261] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.908206 22206 consensus_queue.cc:237] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } }
I20240502 08:19:21.911294 23135 raft_consensus.cc:1261] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.911877 23846 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.915102 23843 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.915583 22337 raft_consensus.cc:1261] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.916031 23617 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.916858 23811 raft_consensus.cc:1050] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a to VOTER
I20240502 08:19:21.917121 23135 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.917676 23811 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:21.918303 23843 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.920603 21923 catalog_manager.cc:5484] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.921792 21918 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:21.922684 22206 raft_consensus.cc:1261] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.926786 23398 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.926652 21923 catalog_manager.cc:5484] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:21.927489 23809 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.928097 22206 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.928876 22070 raft_consensus.cc:1261] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.931221 23811 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.931668 22318 tablet_service.cc:1514] Processing DeleteTablet for tablet cd3b3b39b7b54ef3b7c3b5e7265b76de with delete_type TABLET_DATA_TOMBSTONED (TS ecc8a9127fcc4e3a8befd1e0ddfebeb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39822
I20240502 08:19:21.931991 22206 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.932567 22072 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:21.933100 23811 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:21.924803 23928 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20240502 08:19:21.939293 23267 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.952049 23929 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.932000 23846 consensus_queue.cc:1035] T cd3b3b39b7b54ef3b7c3b5e7265b76de P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20240502 08:19:21.955626 22071 raft_consensus.cc:2932] T cd3b3b39b7b54ef3b7c3b5e7265b76de P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.958110 22337 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.958707 22070 consensus_queue.cc:237] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:21.959506 23934 tablet_replica.cc:330] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: stopping tablet replica
I20240502 08:19:21.960006 23934 raft_consensus.cc:2229] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:21.961163 23934 raft_consensus.cc:2258] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:21.963223 23811 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.963531 21930 catalog_manager.cc:5484] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.964084 23267 raft_consensus.cc:1261] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.965270 23135 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.967772 22337 raft_consensus.cc:1261] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.967994 22205 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.968156 23398 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.969313 22996 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.972154 22996 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.972951 22203 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.973843 21930 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:21.980460 23934 ts_tablet_manager.cc:1875] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:21.981163 22072 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:21.982414 23266 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.983546 22071 consensus_queue.cc:237] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } }
I20240502 08:19:21.985206 23844 consensus_queue.cc:1035] T 5507342ac56b42878ce02a2dcf149bc0 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.987897 22203 raft_consensus.cc:1261] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.988444 23927 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:21.992687 22337 raft_consensus.cc:2932] T 5507342ac56b42878ce02a2dcf149bc0 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.992862 21926 catalog_manager.cc:5484] T 5507342ac56b42878ce02a2dcf149bc0 P 26d5ad8a5f594649b73ab2b11dc3d328 reported cstate change: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:21.993690 23934 ts_tablet_manager.cc:1888] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:21.993875 23934 log.cc:1196] T cd3b3b39b7b54ef3b7c3b5e7265b76de P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/cd3b3b39b7b54ef3b7c3b5e7265b76de
I20240502 08:19:21.996131 23136 raft_consensus.cc:1261] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Refusing update from remote peer d24c3c61ac4a4eefa0e4a5d9cef079b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:21.997427 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 5507342ac56b42878ce02a2dcf149bc0 with delete_type TABLET_DATA_TOMBSTONED (TS 8ef0b298bd6642ea900f3e9843bc3031 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39008
I20240502 08:19:21.997825 23895 tablet_replica.cc:330] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: stopping tablet replica
I20240502 08:19:21.998035 23895 raft_consensus.cc:2229] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:21.998281 23895 raft_consensus.cc:2258] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:21.998360 23844 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:21.999826 21916 catalog_manager.cc:4830] TS ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709): tablet cd3b3b39b7b54ef3b7c3b5e7265b76de (table rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]) successfully deleted
I20240502 08:19:22.000033 23928 consensus_queue.cc:1035] T f2122b14ffc2475cb8a8a18ba789f68d P d24c3c61ac4a4eefa0e4a5d9cef079b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:22.002838 23895 ts_tablet_manager.cc:1875] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:22.002533 23136 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:22.004031 22205 raft_consensus.cc:2932] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:22.009119 21918 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f2122b14ffc2475cb8a8a18ba789f68d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:22.009243 21923 catalog_manager.cc:5484] T f2122b14ffc2475cb8a8a18ba789f68d P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 5 to 6, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:22.013171 23895 ts_tablet_manager.cc:1888] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:22.013389 23895 log.cc:1196] T 5507342ac56b42878ce02a2dcf149bc0 P 8ef0b298bd6642ea900f3e9843bc3031: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/5507342ac56b42878ce02a2dcf149bc0
I20240502 08:19:22.014549 21918 catalog_manager.cc:4830] TS 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597): tablet 5507342ac56b42878ce02a2dcf149bc0 (table rebalance_test_table_0 [id=91ff6682ef8344febde681488109259d]) successfully deleted
I20240502 08:19:22.014823 21918 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5507342ac56b42878ce02a2dcf149bc0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:22.015492 22337 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:22.017206 22318 tablet_service.cc:1514] Processing DeleteTablet for tablet f2122b14ffc2475cb8a8a18ba789f68d with delete_type TABLET_DATA_TOMBSTONED (TS ecc8a9127fcc4e3a8befd1e0ddfebeb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39822
I20240502 08:19:22.017572 23934 tablet_replica.cc:330] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: stopping tablet replica
I20240502 08:19:22.017822 23934 raft_consensus.cc:2229] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:22.018163 23934 raft_consensus.cc:2258] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:22.018375 23398 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:22.018970 23811 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:22.019322 23934 ts_tablet_manager.cc:1875] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:22.019834 22205 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:22.020435 23811 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:22.021529 23809 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.024655 23397 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.025832 23934 ts_tablet_manager.cc:1888] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:22.025863 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82db7fca82b5492faa762066294de080 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:22.025950 23934 log.cc:1196] T f2122b14ffc2475cb8a8a18ba789f68d P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/f2122b14ffc2475cb8a8a18ba789f68d
I20240502 08:19:22.026711 21916 catalog_manager.cc:4830] TS ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709): tablet f2122b14ffc2475cb8a8a18ba789f68d (table rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]) successfully deleted
I20240502 08:19:22.026964 22205 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.028115 23809 raft_consensus.cc:1050] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 to VOTER
I20240502 08:19:22.028643 23809 consensus_queue.cc:237] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:22.030598 23266 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:22.031222 23812 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:22.031672 22203 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:22.031546 21925 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:22.032403 23812 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:22.043690 22050 tablet_service.cc:1514] Processing DeleteTablet for tablet 82db7fca82b5492faa762066294de080 with delete_type TABLET_DATA_TOMBSTONED (TS d24c3c61ac4a4eefa0e4a5d9cef079b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43852
I20240502 08:19:22.044116 23911 tablet_replica.cc:330] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: stopping tablet replica
I20240502 08:19:22.044380 23911 raft_consensus.cc:2229] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:22.044822 23911 raft_consensus.cc:2258] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:22.046298 23911 ts_tablet_manager.cc:1875] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:22.049350 22072 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:22.050154 23809 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:22.059193 23911 ts_tablet_manager.cc:1888] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:22.062970 23911 log.cc:1196] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/82db7fca82b5492faa762066294de080
I20240502 08:19:22.063910 21918 catalog_manager.cc:4830] TS d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747): tablet 82db7fca82b5492faa762066294de080 (table rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]) successfully deleted
I20240502 08:19:22.067169 23809 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.068542 23266 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.071578 22072 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.073060 21925 catalog_manager.cc:5484] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 reported cstate change: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.077286 22203 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.086040 22337 consensus_queue.cc:237] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:22.087884 22072 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:22.088012 23266 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:22.088759 23812 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:22.089434 23811 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:22.091418 23812 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.092687 23266 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.094193 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74f2e5440fd949b489858fd52250f668 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:22.094798 22072 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:22.098023 21927 catalog_manager.cc:5484] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 5 to 6, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:22.102941 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f2e5440fd949b489858fd52250f668 with delete_type TABLET_DATA_TOMBSTONED (TS 8ef0b298bd6642ea900f3e9843bc3031 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39008
I20240502 08:19:22.103303 23895 tablet_replica.cc:330] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: stopping tablet replica
I20240502 08:19:22.103500 23895 raft_consensus.cc:2229] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:22.103840 23895 raft_consensus.cc:2258] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:22.105077 23895 ts_tablet_manager.cc:1875] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:22.110574 23895 ts_tablet_manager.cc:1888] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:22.110813 23895 log.cc:1196] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/74f2e5440fd949b489858fd52250f668
I20240502 08:19:22.111553 21918 catalog_manager.cc:4830] TS 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597): tablet 74f2e5440fd949b489858fd52250f668 (table rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]) successfully deleted
I20240502 08:19:22.127081 23846 raft_consensus.cc:1050] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: attempting to promote NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a to VOTER
I20240502 08:19:22.127954 23846 consensus_queue.cc:237] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:22.130313 23397 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:22.130916 23845 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:22.131542 22337 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:22.137262 23846 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:22.138020 22072 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:22.138731 23617 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:22.151000 23617 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.153573 22072 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.156978 22337 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.160823 21926 catalog_manager.cc:5484] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.172052 23397 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.176278 22203 consensus_queue.cc:237] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:22.178540 22337 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:22.178834 23397 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:22.179194 23617 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:22.179554 23974 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:22.182225 23974 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.183089 23397 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.186857 21918 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cd539305b35444596c668a83ace2fbd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:22.187635 22337 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.189170 21926 catalog_manager.cc:5484] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a reported cstate change: config changed from index 5 to 6, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:22.193921 22050 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd539305b35444596c668a83ace2fbd with delete_type TABLET_DATA_TOMBSTONED (TS d24c3c61ac4a4eefa0e4a5d9cef079b8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43852
I20240502 08:19:22.194343 23911 tablet_replica.cc:330] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: stopping tablet replica
I20240502 08:19:22.194541 23911 raft_consensus.cc:2229] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:22.194909 23911 raft_consensus.cc:2258] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:22.195942 23911 ts_tablet_manager.cc:1875] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:22.201691 23911 ts_tablet_manager.cc:1888] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:22.201819 23911 log.cc:1196] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/4cd539305b35444596c668a83ace2fbd
I20240502 08:19:22.202626 21918 catalog_manager.cc:4830] TS d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747): tablet 4cd539305b35444596c668a83ace2fbd (table rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]) successfully deleted
I20240502 08:19:22.305078 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.313107 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:22.324945 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:22.325759 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:22.336180 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.372138 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.531996 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.534055 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:22.547376 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.551807 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.567373 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:22.570878 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:22.677953 22203 consensus_queue.cc:237] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } }
I20240502 08:19:22.679955 23397 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:22.680083 22337 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:22.680606 23845 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.681138 23845 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:19:22.681936 22141 consensus_peers.cc:485] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 -> Peer d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747): Couldn't send request to peer d24c3c61ac4a4eefa0e4a5d9cef079b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.683938 23845 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } } }
I20240502 08:19:22.684510 22337 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } } }
I20240502 08:19:22.685319 23397 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } } }
I20240502 08:19:22.689618 21926 catalog_manager.cc:5484] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:22.724442 22337 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } }
I20240502 08:19:22.726820 23397 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:19:22.727279 22274 consensus_peers.cc:485] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747): Couldn't send request to peer d24c3c61ac4a4eefa0e4a5d9cef079b8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.727778 23809 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.730943 23809 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } } }
I20240502 08:19:22.734781 22203 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:22.735380 23397 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } } }
I20240502 08:19:22.735549 21923 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:22.735987 24090 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.737506 22203 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } } }
I20240502 08:19:22.756561 22337 consensus_queue.cc:237] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: true } }
I20240502 08:19:22.758694 23266 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:22.758919 22072 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:19:22.759120 22274 consensus_peers.cc:485] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597): Couldn't send request to peer 8ef0b298bd6642ea900f3e9843bc3031. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.759548 24090 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.759960 23809 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.762574 24090 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: true } } }
I20240502 08:19:22.763029 22072 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: true } } }
I20240502 08:19:22.768211 23266 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: true } } }
I20240502 08:19:22.768863 21927 catalog_manager.cc:5484] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: config changed from index 6 to 7, NON_VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: true } } }
I20240502 08:19:22.830032 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.848387 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.857100 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:22.857951 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:22.859246 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:22.871646 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:22.949702 22337 consensus_queue.cc:237] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:22.952250 23397 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:22.952874 22203 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:22.952868 24101 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.953419 24090 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.956485 24090 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:22.957219 22203 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:22.958076 23397 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
I20240502 08:19:22.962873 21927 catalog_manager.cc:5484] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 6 to 7, NON_VOTER fdea103695a94a099acd05c42bcccdca (127.6.151.70) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } } }
W20240502 08:19:22.963972 22272 consensus_peers.cc:485] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer fdea103695a94a099acd05c42bcccdca (127.6.151.70:46149): Couldn't send request to peer fdea103695a94a099acd05c42bcccdca. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8069f943c7234829b59c930515491ad5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.977458 22337 consensus_queue.cc:237] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:22.979964 22072 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:22.980023 23266 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:19:22.980124 22274 consensus_peers.cc:485] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181): Couldn't send request to peer b6ca0fbc79ad4b1780eebe8bc88c733a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 169d28e2ca504bcfb757b049d7aa1bee. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:22.980573 24101 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.980855 23809 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:22.983705 23809 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:22.984321 22072 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:22.988433 23266 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:22.989005 21927 catalog_manager.cc:5484] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: config changed from index 6 to 7, NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } } }
I20240502 08:19:23.163838 24177 ts_tablet_manager.cc:902] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:23.165386 24177 tablet_copy_client.cc:308] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:23.166342 22358 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 82db7fca82b5492faa762066294de080 from peer d24c3c61ac4a4eefa0e4a5d9cef079b8 ({username='slave'} at 127.6.151.65:40559)
I20240502 08:19:23.166496 22358 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 82db7fca82b5492faa762066294de080 from peer d24c3c61ac4a4eefa0e4a5d9cef079b8 at {username='slave'} at 127.6.151.65:40559: session id = d24c3c61ac4a4eefa0e4a5d9cef079b8-82db7fca82b5492faa762066294de080
I20240502 08:19:23.168645 22358 tablet_copy_source_session.cc:205] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.170040 24177 ts_tablet_manager.cc:1875] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:19:23.175046 24177 ts_tablet_manager.cc:1888] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:19:23.175329 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82db7fca82b5492faa762066294de080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.177824 24177 tablet_copy_client.cc:783] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.178153 24177 tablet_copy_client.cc:647] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.180248 24177 tablet_copy_client.cc:523] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.183223 24177 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:23.208732 24183 ts_tablet_manager.cc:902] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:23.210546 24183 tablet_copy_client.cc:308] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:23.211781 22358 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 74f2e5440fd949b489858fd52250f668 from peer 8ef0b298bd6642ea900f3e9843bc3031 ({username='slave'} at 127.6.151.66:58203)
I20240502 08:19:23.211954 22358 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 74f2e5440fd949b489858fd52250f668 from peer 8ef0b298bd6642ea900f3e9843bc3031 at {username='slave'} at 127.6.151.66:58203: session id = 8ef0b298bd6642ea900f3e9843bc3031-74f2e5440fd949b489858fd52250f668
I20240502 08:19:23.214004 22358 tablet_copy_source_session.cc:205] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.215365 24183 ts_tablet_manager.cc:1875] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:19:23.220178 24183 ts_tablet_manager.cc:1888] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:19:23.220499 24183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74f2e5440fd949b489858fd52250f668. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.220997 24177 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.221436 24177 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap complete.
I20240502 08:19:23.221721 24177 ts_tablet_manager.cc:1367] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.039s	user 0.031s	sys 0.004s
I20240502 08:19:23.222532 24177 raft_consensus.cc:348] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } }
I20240502 08:19:23.222805 24177 raft_consensus.cc:729] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: LEARNER
I20240502 08:19:23.223002 24177 consensus_queue.cc:260] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } }
I20240502 08:19:23.223147 24183 tablet_copy_client.cc:783] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.223832 24177 ts_tablet_manager.cc:1398] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 08:19:23.224128 24183 tablet_copy_client.cc:647] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.225585 22358 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session d24c3c61ac4a4eefa0e4a5d9cef079b8-82db7fca82b5492faa762066294de080 received from {username='slave'} at 127.6.151.65:40559
I20240502 08:19:23.225739 22358 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session d24c3c61ac4a4eefa0e4a5d9cef079b8-82db7fca82b5492faa762066294de080 on tablet 82db7fca82b5492faa762066294de080 with peer d24c3c61ac4a4eefa0e4a5d9cef079b8
I20240502 08:19:23.227708 24183 tablet_copy_client.cc:523] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.233623 24183 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap starting.
I20240502 08:19:23.279333 24177 ts_tablet_manager.cc:902] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Initiating tablet copy from peer 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597)
I20240502 08:19:23.283041 24177 tablet_copy_client.cc:308] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:42597
I20240502 08:19:23.283885 22226 tablet_copy_service.cc:140] P 8ef0b298bd6642ea900f3e9843bc3031: Received BeginTabletCopySession request for tablet 4cd539305b35444596c668a83ace2fbd from peer d24c3c61ac4a4eefa0e4a5d9cef079b8 ({username='slave'} at 127.6.151.65:38625)
I20240502 08:19:23.284062 22226 tablet_copy_service.cc:161] P 8ef0b298bd6642ea900f3e9843bc3031: Beginning new tablet copy session on tablet 4cd539305b35444596c668a83ace2fbd from peer d24c3c61ac4a4eefa0e4a5d9cef079b8 at {username='slave'} at 127.6.151.65:38625: session id = d24c3c61ac4a4eefa0e4a5d9cef079b8-4cd539305b35444596c668a83ace2fbd
I20240502 08:19:23.284956 24183 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.285459 24183 tablet_bootstrap.cc:492] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Bootstrap complete.
I20240502 08:19:23.285758 24183 ts_tablet_manager.cc:1367] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent bootstrapping tablet: real 0.052s	user 0.051s	sys 0.000s
I20240502 08:19:23.286515 22226 tablet_copy_source_session.cc:205] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.286510 24183 raft_consensus.cc:348] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: true } }
I20240502 08:19:23.286808 24183 raft_consensus.cc:729] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ef0b298bd6642ea900f3e9843bc3031, State: Initialized, Role: LEARNER
I20240502 08:19:23.287043 24183 consensus_queue.cc:260] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: NON_VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: true } }
I20240502 08:19:23.288089 24177 ts_tablet_manager.cc:1875] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:19:23.288159 24183 ts_tablet_manager.cc:1398] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031: Time spent starting tablet: real 0.002s	user 0.005s	sys 0.000s
I20240502 08:19:23.289363 22357 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session 8ef0b298bd6642ea900f3e9843bc3031-74f2e5440fd949b489858fd52250f668 received from {username='slave'} at 127.6.151.66:58203
I20240502 08:19:23.289512 22357 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session 8ef0b298bd6642ea900f3e9843bc3031-74f2e5440fd949b489858fd52250f668 on tablet 74f2e5440fd949b489858fd52250f668 with peer 8ef0b298bd6642ea900f3e9843bc3031
I20240502 08:19:23.295871 24177 ts_tablet_manager.cc:1888] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:19:23.296192 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cd539305b35444596c668a83ace2fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.299027 24177 tablet_copy_client.cc:783] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.299263 24177 tablet_copy_client.cc:647] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.301056 24177 tablet_copy_client.cc:523] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.303897 24177 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap starting.
I20240502 08:19:23.334378 24177 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.334805 24177 tablet_bootstrap.cc:492] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Bootstrap complete.
I20240502 08:19:23.335023 24177 ts_tablet_manager.cc:1367] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent bootstrapping tablet: real 0.031s	user 0.028s	sys 0.004s
I20240502 08:19:23.335667 24177 raft_consensus.cc:348] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } }
I20240502 08:19:23.335877 24177 raft_consensus.cc:729] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d24c3c61ac4a4eefa0e4a5d9cef079b8, State: Initialized, Role: LEARNER
I20240502 08:19:23.336058 24177 consensus_queue.cc:260] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: NON_VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: true } }
I20240502 08:19:23.336479 24177 ts_tablet_manager.cc:1398] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:23.337342 22226 tablet_copy_service.cc:342] P 8ef0b298bd6642ea900f3e9843bc3031: Request end of tablet copy session d24c3c61ac4a4eefa0e4a5d9cef079b8-4cd539305b35444596c668a83ace2fbd received from {username='slave'} at 127.6.151.65:38625
I20240502 08:19:23.337488 22226 tablet_copy_service.cc:433] P 8ef0b298bd6642ea900f3e9843bc3031: ending tablet copy session d24c3c61ac4a4eefa0e4a5d9cef079b8-4cd539305b35444596c668a83ace2fbd on tablet 4cd539305b35444596c668a83ace2fbd with peer d24c3c61ac4a4eefa0e4a5d9cef079b8
I20240502 08:19:23.490279 24230 ts_tablet_manager.cc:902] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:23.490481 24231 ts_tablet_manager.cc:902] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:23.492089 24230 tablet_copy_client.cc:308] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:23.492703 24231 tablet_copy_client.cc:308] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:23.492949 22357 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 169d28e2ca504bcfb757b049d7aa1bee from peer b6ca0fbc79ad4b1780eebe8bc88c733a ({username='slave'} at 127.6.151.68:45023)
I20240502 08:19:23.493116 22357 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 169d28e2ca504bcfb757b049d7aa1bee from peer b6ca0fbc79ad4b1780eebe8bc88c733a at {username='slave'} at 127.6.151.68:45023: session id = b6ca0fbc79ad4b1780eebe8bc88c733a-169d28e2ca504bcfb757b049d7aa1bee
I20240502 08:19:23.496037 22357 tablet_copy_source_session.cc:205] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.497480 24230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169d28e2ca504bcfb757b049d7aa1bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.502393 24230 tablet_copy_client.cc:783] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.502497 22357 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 8069f943c7234829b59c930515491ad5 from peer fdea103695a94a099acd05c42bcccdca ({username='slave'} at 127.6.151.70:53775)
I20240502 08:19:23.502678 24230 tablet_copy_client.cc:647] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.502723 22357 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 8069f943c7234829b59c930515491ad5 from peer fdea103695a94a099acd05c42bcccdca at {username='slave'} at 127.6.151.70:53775: session id = fdea103695a94a099acd05c42bcccdca-8069f943c7234829b59c930515491ad5
I20240502 08:19:23.504500 24230 tablet_copy_client.cc:523] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.505851 22357 tablet_copy_source_session.cc:205] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:23.507578 24231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8069f943c7234829b59c930515491ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:23.509455 24230 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap starting.
I20240502 08:19:23.515022 24231 tablet_copy_client.cc:783] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:23.515298 24231 tablet_copy_client.cc:647] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:23.517072 24231 tablet_copy_client.cc:523] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:23.521035 24231 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: Bootstrap starting.
I20240502 08:19:23.564388 24230 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.564886 24230 tablet_bootstrap.cc:492] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: Bootstrap complete.
I20240502 08:19:23.565218 24230 ts_tablet_manager.cc:1367] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent bootstrapping tablet: real 0.056s	user 0.047s	sys 0.007s
I20240502 08:19:23.566043 24230 raft_consensus.cc:348] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:23.566330 24230 raft_consensus.cc:729] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b6ca0fbc79ad4b1780eebe8bc88c733a, State: Initialized, Role: LEARNER
I20240502 08:19:23.566594 24230 consensus_queue.cc:260] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: true } }
I20240502 08:19:23.568145 24230 ts_tablet_manager.cc:1398] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
I20240502 08:19:23.569115 24231 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:23.569629 24231 tablet_bootstrap.cc:492] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: Bootstrap complete.
I20240502 08:19:23.569922 22357 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-169d28e2ca504bcfb757b049d7aa1bee received from {username='slave'} at 127.6.151.68:45023
I20240502 08:19:23.569988 24231 ts_tablet_manager.cc:1367] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: Time spent bootstrapping tablet: real 0.049s	user 0.043s	sys 0.004s
I20240502 08:19:23.570056 22357 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session b6ca0fbc79ad4b1780eebe8bc88c733a-169d28e2ca504bcfb757b049d7aa1bee on tablet 169d28e2ca504bcfb757b049d7aa1bee with peer b6ca0fbc79ad4b1780eebe8bc88c733a
I20240502 08:19:23.570834 24231 raft_consensus.cc:348] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:23.571159 24231 raft_consensus.cc:729] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdea103695a94a099acd05c42bcccdca, State: Initialized, Role: LEARNER
I20240502 08:19:23.571458 24231 consensus_queue.cc:260] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: NON_VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: true } }
I20240502 08:19:23.572463 24231 ts_tablet_manager.cc:1398] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:23.574169 22357 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session fdea103695a94a099acd05c42bcccdca-8069f943c7234829b59c930515491ad5 received from {username='slave'} at 127.6.151.70:53775
I20240502 08:19:23.574296 22357 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session fdea103695a94a099acd05c42bcccdca-8069f943c7234829b59c930515491ad5 on tablet 8069f943c7234829b59c930515491ad5 with peer fdea103695a94a099acd05c42bcccdca
I20240502 08:19:23.680616 22072 raft_consensus.cc:1203] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:19:23.685864 22203 raft_consensus.cc:1203] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:19:23.737829 22072 raft_consensus.cc:1203] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:19:24.018325 23135 raft_consensus.cc:1203] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:19:24.027851 23397 raft_consensus.cc:1203] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7]   Dedup: 1.7->[]
I20240502 08:19:24.099766 24261 raft_consensus.cc:1050] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031: attempting to promote NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 to VOTER
I20240502 08:19:24.100505 24261 consensus_queue.cc:237] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } }
I20240502 08:19:24.103977 22072 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.104015 23397 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.104063 22337 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.104602 24315 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.105119 24314 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.105501 23846 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.109265 24314 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.110152 22337 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.113905 23397 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.114514 21926 catalog_manager.cc:5484] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:24.125696 22072 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } attrs { replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.130957 22205 consensus_queue.cc:237] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } }
I20240502 08:19:24.134754 22072 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.134835 23397 raft_consensus.cc:1261] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer 8ef0b298bd6642ea900f3e9843bc3031: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.135469 24261 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.135955 24261 consensus_queue.cc:1035] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.138398 24261 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.141081 22071 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.146538 21918 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4cd539305b35444596c668a83ace2fbd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:24.148900 21927 catalog_manager.cc:5484] T 4cd539305b35444596c668a83ace2fbd P d24c3c61ac4a4eefa0e4a5d9cef079b8 reported cstate change: config changed from index 8 to 9, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "8ef0b298bd6642ea900f3e9843bc3031" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.148983 23397 raft_consensus.cc:2932] T 4cd539305b35444596c668a83ace2fbd P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.153769 22318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cd539305b35444596c668a83ace2fbd with delete_type TABLET_DATA_TOMBSTONED (TS ecc8a9127fcc4e3a8befd1e0ddfebeb3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39822
I20240502 08:19:24.158484 23809 raft_consensus.cc:1050] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER 8ef0b298bd6642ea900f3e9843bc3031 to VOTER
I20240502 08:19:24.159718 23809 consensus_queue.cc:237] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } }
I20240502 08:19:24.162786 22205 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.162871 22071 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.162884 23266 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.163475 24262 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.163846 24090 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.164404 24090 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.172814 24330 tablet_replica.cc:330] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: stopping tablet replica
I20240502 08:19:24.173079 24330 raft_consensus.cc:2229] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:24.173506 24330 raft_consensus.cc:2258] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:24.177418 24330 ts_tablet_manager.cc:1875] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:24.180287 23809 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.181872 23267 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.183972 22203 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.191876 21927 catalog_manager.cc:5484] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 7 to 8, 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.193607 22071 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.199812 22337 consensus_queue.cc:237] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } }
I20240502 08:19:24.203596 23267 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.203882 24330 ts_tablet_manager.cc:1888] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:19:24.204094 24330 log.cc:1196] T 4cd539305b35444596c668a83ace2fbd P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/4cd539305b35444596c668a83ace2fbd
I20240502 08:19:24.204169 22205 raft_consensus.cc:1261] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.204341 24262 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.204861 24090 consensus_queue.cc:1035] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.205399 21916 catalog_manager.cc:4830] TS ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709): tablet 4cd539305b35444596c668a83ace2fbd (table rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]) successfully deleted
I20240502 08:19:24.207342 24090 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.207798 22205 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.208222 23266 raft_consensus.cc:2932] T 74f2e5440fd949b489858fd52250f668 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.211251 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74f2e5440fd949b489858fd52250f668 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:24.212330 21925 catalog_manager.cc:5484] T 74f2e5440fd949b489858fd52250f668 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 8 to 9, VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { promote: false } } }
I20240502 08:19:24.216104 22050 tablet_service.cc:1514] Processing DeleteTablet for tablet 74f2e5440fd949b489858fd52250f668 with delete_type TABLET_DATA_TOMBSTONED (TS d24c3c61ac4a4eefa0e4a5d9cef079b8 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43852
I20240502 08:19:24.217132 24340 tablet_replica.cc:330] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: stopping tablet replica
I20240502 08:19:24.217342 24340 raft_consensus.cc:2229] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:24.217662 24340 raft_consensus.cc:2258] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:24.218580 24340 ts_tablet_manager.cc:1875] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:24.223685 24340 ts_tablet_manager.cc:1888] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:19:24.223822 24340 log.cc:1196] T 74f2e5440fd949b489858fd52250f668 P d24c3c61ac4a4eefa0e4a5d9cef079b8: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/74f2e5440fd949b489858fd52250f668
I20240502 08:19:24.224611 21918 catalog_manager.cc:4830] TS d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747): tablet 74f2e5440fd949b489858fd52250f668 (table rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]) successfully deleted
I20240502 08:19:24.234431 24262 raft_consensus.cc:1050] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER d24c3c61ac4a4eefa0e4a5d9cef079b8 to VOTER
I20240502 08:19:24.235183 24262 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } }
I20240502 08:19:24.237648 22071 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.237650 23397 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.238268 24262 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.238574 24333 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.238929 22205 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.239524 24262 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.242847 24333 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.243671 23397 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.244433 22205 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.249578 21927 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a reported cstate change: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.250587 22071 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } attrs { replace: true } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.255271 22337 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } }
I20240502 08:19:24.257575 22071 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.257593 23397 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.258442 23809 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.258787 24090 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.262540 24262 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.263255 23397 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.266175 22071 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.268640 21925 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a reported cstate change: config changed from index 8 to 9, VOTER 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } }
I20240502 08:19:24.270188 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82db7fca82b5492faa762066294de080 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:24.272809 22184 tablet_service.cc:1514] Processing DeleteTablet for tablet 82db7fca82b5492faa762066294de080 with delete_type TABLET_DATA_TOMBSTONED (TS 8ef0b298bd6642ea900f3e9843bc3031 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39008
I20240502 08:19:24.274250 24358 tablet_replica.cc:330] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: stopping tablet replica
I20240502 08:19:24.274463 24358 raft_consensus.cc:2229] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:24.275247 24358 raft_consensus.cc:2258] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:24.276226 24358 ts_tablet_manager.cc:1875] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:24.280794 24358 ts_tablet_manager.cc:1888] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:19:24.280913 24358 log.cc:1196] T 82db7fca82b5492faa762066294de080 P 8ef0b298bd6642ea900f3e9843bc3031: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/82db7fca82b5492faa762066294de080
I20240502 08:19:24.281728 21918 catalog_manager.cc:4830] TS 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597): tablet 82db7fca82b5492faa762066294de080 (table rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]) successfully deleted
I20240502 08:19:24.371141 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:24.376801 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:24.404706 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:24.405603 24262 raft_consensus.cc:1050] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER fdea103695a94a099acd05c42bcccdca to VOTER
I20240502 08:19:24.406525 24262 consensus_queue.cc:237] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } }
I20240502 08:19:24.417071 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:24.425891 23135 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.426735 24090 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.429791 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:24.434386 23397 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.439742 22205 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.440358 24090 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.441181 24090 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.467202 24090 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.471621 22205 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.477313 21923 catalog_manager.cc:5484] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 7 to 8, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.478570 23397 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.485308 23136 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdea103695a94a099acd05c42bcccdca (127.6.151.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.489084 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:24.491607 22337 consensus_queue.cc:237] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } }
I20240502 08:19:24.493935 22205 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.494606 24262 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.497439 24262 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.499791 22205 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.505906 23136 raft_consensus.cc:1261] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.506685 24333 consensus_queue.cc:1035] T 8069f943c7234829b59c930515491ad5 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.508749 23136 raft_consensus.cc:2932] T 8069f943c7234829b59c930515491ad5 P fdea103695a94a099acd05c42bcccdca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.509712 21923 catalog_manager.cc:5484] T 8069f943c7234829b59c930515491ad5 P 8ef0b298bd6642ea900f3e9843bc3031 reported cstate change: config changed from index 8 to 9, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ef0b298bd6642ea900f3e9843bc3031" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 42597 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "fdea103695a94a099acd05c42bcccdca" member_type: VOTER last_known_addr { host: "127.6.151.70" port: 46149 } attrs { promote: false } } }
I20240502 08:19:24.511197 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8069f943c7234829b59c930515491ad5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:24.551188 23378 tablet_service.cc:1514] Processing DeleteTablet for tablet 8069f943c7234829b59c930515491ad5 with delete_type TABLET_DATA_TOMBSTONED (TS b6ca0fbc79ad4b1780eebe8bc88c733a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35282
I20240502 08:19:24.558552 24408 tablet_replica.cc:330] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: stopping tablet replica
I20240502 08:19:24.559230 24408 raft_consensus.cc:2229] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:24.560107 24408 raft_consensus.cc:2258] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:24.561650 24408 ts_tablet_manager.cc:1875] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:24.574369 24408 ts_tablet_manager.cc:1888] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:19:24.574659 24408 log.cc:1196] T 8069f943c7234829b59c930515491ad5 P b6ca0fbc79ad4b1780eebe8bc88c733a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/8069f943c7234829b59c930515491ad5
I20240502 08:19:24.575562 21918 catalog_manager.cc:4830] TS b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181): tablet 8069f943c7234829b59c930515491ad5 (table rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]) successfully deleted
I20240502 08:19:24.596688 24090 raft_consensus.cc:1050] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER b6ca0fbc79ad4b1780eebe8bc88c733a to VOTER
I20240502 08:19:24.597430 24090 consensus_queue.cc:237] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:24.599864 23397 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.599920 23266 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.599932 22071 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:24.600478 23809 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.600807 24090 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.601090 24333 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:24.604589 23809 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:24.605258 22071 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:24.606844 23397 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:24.608476 23266 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:24.609555 21923 catalog_manager.cc:5484] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 7 to 8, b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:24.614434 22337 consensus_queue.cc:237] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } }
I20240502 08:19:24.616420 22071 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.616420 23397 raft_consensus.cc:1261] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:24.616899 23809 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.617197 24262 consensus_queue.cc:1035] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:24.622174 24090 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:24.625705 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 169d28e2ca504bcfb757b049d7aa1bee with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:24.626953 22071 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:24.626932 23397 raft_consensus.cc:2932] T 169d28e2ca504bcfb757b049d7aa1bee P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } } }
I20240502 08:19:24.627497 21923 catalog_manager.cc:5484] T 169d28e2ca504bcfb757b049d7aa1bee P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 8 to 9, VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:24.641340 23247 tablet_service.cc:1514] Processing DeleteTablet for tablet 169d28e2ca504bcfb757b049d7aa1bee with delete_type TABLET_DATA_TOMBSTONED (TS 26d5ad8a5f594649b73ab2b11dc3d328 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33820
I20240502 08:19:24.642402 24419 tablet_replica.cc:330] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: stopping tablet replica
I20240502 08:19:24.642864 24419 raft_consensus.cc:2229] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:24.643304 24419 raft_consensus.cc:2258] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:24.644604 24419 ts_tablet_manager.cc:1875] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:24.649478 24419 ts_tablet_manager.cc:1888] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:19:24.649638 24419 log.cc:1196] T 169d28e2ca504bcfb757b049d7aa1bee P 26d5ad8a5f594649b73ab2b11dc3d328: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/wals/169d28e2ca504bcfb757b049d7aa1bee
I20240502 08:19:24.650539 21916 catalog_manager.cc:4830] TS 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69:41139): tablet 169d28e2ca504bcfb757b049d7aa1bee (table rebalance_test_table_2 [id=e94b80c7f18e4a22b6ee6022e68fc0e3]) successfully deleted
I20240502 08:19:24.876317 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:24.879024 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:24.910133 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:24.910759 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:24.911084 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:24.911091 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:24.994876 22337 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:24.997015 22071 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 08:19:24.997133 22272 consensus_peers.cc:485] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69:41139): Couldn't send request to peer 26d5ad8a5f594649b73ab2b11dc3d328. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82db7fca82b5492faa762066294de080. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:24.997359 23397 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:24.997710 23809 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:24.998049 24090 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:25.000846 23809 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:25.001510 23397 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:25.001533 22071 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } } }
I20240502 08:19:25.007673 21925 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 9 to 10, NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:25.514297 24474 ts_tablet_manager.cc:902] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: Initiating tablet copy from peer ecc8a9127fcc4e3a8befd1e0ddfebeb3 (127.6.151.67:41709)
I20240502 08:19:25.517236 24474 tablet_copy_client.cc:308] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:41709
I20240502 08:19:25.518174 22357 tablet_copy_service.cc:140] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Received BeginTabletCopySession request for tablet 82db7fca82b5492faa762066294de080 from peer 26d5ad8a5f594649b73ab2b11dc3d328 ({username='slave'} at 127.6.151.69:35553)
I20240502 08:19:25.518383 22357 tablet_copy_service.cc:161] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Beginning new tablet copy session on tablet 82db7fca82b5492faa762066294de080 from peer 26d5ad8a5f594649b73ab2b11dc3d328 at {username='slave'} at 127.6.151.69:35553: session id = 26d5ad8a5f594649b73ab2b11dc3d328-82db7fca82b5492faa762066294de080
I20240502 08:19:25.520313 22357 tablet_copy_source_session.cc:205] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:25.521505 24474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82db7fca82b5492faa762066294de080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:25.526173 24474 tablet_copy_client.cc:783] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:25.526432 24474 tablet_copy_client.cc:647] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:25.528497 24474 tablet_copy_client.cc:523] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:25.531217 24474 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap starting.
I20240502 08:19:25.565196 24474 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:25.565578 24474 tablet_bootstrap.cc:492] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: Bootstrap complete.
I20240502 08:19:25.565819 24474 ts_tablet_manager.cc:1367] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent bootstrapping tablet: real 0.035s	user 0.029s	sys 0.002s
I20240502 08:19:25.566545 24474 raft_consensus.cc:348] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:25.566869 24474 raft_consensus.cc:729] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 26d5ad8a5f594649b73ab2b11dc3d328, State: Initialized, Role: LEARNER
I20240502 08:19:25.567086 24474 consensus_queue.cc:260] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: true } }
I20240502 08:19:25.568511 24474 ts_tablet_manager.cc:1398] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328: Time spent starting tablet: real 0.003s	user 0.005s	sys 0.000s
I20240502 08:19:25.569350 22357 tablet_copy_service.cc:342] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: Request end of tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-82db7fca82b5492faa762066294de080 received from {username='slave'} at 127.6.151.69:35553
I20240502 08:19:25.569540 22357 tablet_copy_service.cc:433] P ecc8a9127fcc4e3a8befd1e0ddfebeb3: ending tablet copy session 26d5ad8a5f594649b73ab2b11dc3d328-82db7fca82b5492faa762066294de080 on tablet 82db7fca82b5492faa762066294de080 with peer 26d5ad8a5f594649b73ab2b11dc3d328
I20240502 08:19:26.042096 23266 raft_consensus.cc:1203] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10]   Dedup: 1.10->[]
I20240502 08:19:26.641062 24492 raft_consensus.cc:1050] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3: attempting to promote NON_VOTER 26d5ad8a5f594649b73ab2b11dc3d328 to VOTER
I20240502 08:19:26.641786 24492 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:26.644443 22071 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:19:26.644585 23266 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 LEARNER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:19:26.644852 23398 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:19:26.645035 24492 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:19:26.645387 24090 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:19:26.645809 24509 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:19:26.649232 24090 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.650269 22071 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.650509 23398 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.653877 23266 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.654915 21927 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 reported cstate change: config changed from index 10 to 11, 26d5ad8a5f594649b73ab2b11dc3d328 (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6ca0fbc79ad4b1780eebe8bc88c733a" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 34181 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:26.664067 22337 consensus_queue.cc:237] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } }
I20240502 08:19:26.665863 22071 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:19:26.665870 23266 raft_consensus.cc:1261] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Refusing update from remote peer ecc8a9127fcc4e3a8befd1e0ddfebeb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 08:19:26.666394 24090 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:19:26.666702 24492 consensus_queue.cc:1035] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:19:26.669039 24509 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.669858 23266 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.669871 22071 raft_consensus.cc:2932] T 82db7fca82b5492faa762066294de080 P d24c3c61ac4a4eefa0e4a5d9cef079b8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.674113 21916 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82db7fca82b5492faa762066294de080 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:26.674957 21926 catalog_manager.cc:5484] T 82db7fca82b5492faa762066294de080 P 26d5ad8a5f594649b73ab2b11dc3d328 reported cstate change: config changed from index 11 to 12, VOTER b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68) evicted. New cstate: current_term: 1 leader_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ecc8a9127fcc4e3a8befd1e0ddfebeb3" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 41709 } } peers { permanent_uuid: "d24c3c61ac4a4eefa0e4a5d9cef079b8" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 45747 } attrs { promote: false } } peers { permanent_uuid: "26d5ad8a5f594649b73ab2b11dc3d328" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 41139 } attrs { promote: false } } }
I20240502 08:19:26.678829 23378 tablet_service.cc:1514] Processing DeleteTablet for tablet 82db7fca82b5492faa762066294de080 with delete_type TABLET_DATA_TOMBSTONED (TS b6ca0fbc79ad4b1780eebe8bc88c733a not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:35282
I20240502 08:19:26.679488 24519 tablet_replica.cc:330] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: stopping tablet replica
I20240502 08:19:26.679672 24519 raft_consensus.cc:2229] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:26.680078 24519 raft_consensus.cc:2258] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:26.680869 24519 ts_tablet_manager.cc:1875] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:26.685072 24519 ts_tablet_manager.cc:1888] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240502 08:19:26.685204 24519 log.cc:1196] T 82db7fca82b5492faa762066294de080 P b6ca0fbc79ad4b1780eebe8bc88c733a: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.Rebalance_idx3_rf3_t3_l_3_3.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/wals/82db7fca82b5492faa762066294de080
I20240502 08:19:26.685974 21918 catalog_manager.cc:4830] TS b6ca0fbc79ad4b1780eebe8bc88c733a (127.6.151.68:34181): tablet 82db7fca82b5492faa762066294de080 (table rebalance_test_table_1 [id=a59183437a424abba549c03d92c70cdd]) successfully deleted
I20240502 08:19:26.817088 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:26.836405 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:26.849553 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:26.855356 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:26.866324 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:26.879464 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:27.035449 23247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:27.043898 23378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:27.060070 22050 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:27.062803 22318 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:19:27.064671 22184 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:27.101850 23116 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:27.321574  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 21990
I20240502 08:19:27.330371 22113 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:27.480326  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 21990
I20240502 08:19:27.504297  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 22123
I20240502 08:19:27.510835 22245 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 08:19:27.641993 22274 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.151.65:45747: connect: Connection refused (error 111)
W20240502 08:19:27.643429 22274 consensus_peers.cc:485] T 82db7fca82b5492faa762066294de080 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer d24c3c61ac4a4eefa0e4a5d9cef079b8 (127.6.151.65:45747): Couldn't send request to peer d24c3c61ac4a4eefa0e4a5d9cef079b8. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:45747: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:27.658582  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 22123
W20240502 08:19:27.685705 23333 connection.cc:535] server connection from 127.6.151.66:54395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:27.686185 22274 connection.cc:535] client connection to 127.6.151.66:42597 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:19:27.686383  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 22255
W20240502 08:19:27.687337 22274 consensus_peers.cc:485] T 82553085391e47fdae649aa372f54f37 P ecc8a9127fcc4e3a8befd1e0ddfebeb3 -> Peer 8ef0b298bd6642ea900f3e9843bc3031 (127.6.151.66:42597): Couldn't send request to peer 8ef0b298bd6642ea900f3e9843bc3031. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.66:42597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:27.692755 22378 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:27.850158  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 22255
I20240502 08:19:27.877965  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 23315
I20240502 08:19:27.884768 23438 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:28.004607  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 23315
I20240502 08:19:28.030292  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 23185
I20240502 08:19:28.037420 23307 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:28.156808  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 23185
I20240502 08:19:28.178028  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 23054
I20240502 08:19:28.184760 23176 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:28.299975  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 23054
I20240502 08:19:28.321012  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 21899
I20240502 08:19:28.322150 21961 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:28.471936  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 21899
2024-05-02T08:19:28Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.Rebalance/idx3_rf3_t3_l_3_3 (22984 ms)
[ RUN      ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2
I20240502 08:19:28.531250  6749 test_util.cc:268] Using random seed: 1124622084
I20240502 08:19:28.533131  6749 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:19:28.533221  6749 ts_itest-base.cc:113] --------------
I20240502 08:19:28.533286  6749 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:19:28.533360  6749 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:19:28.533421  6749 ts_itest-base.cc:116] --------------
2024-05-02T08:19:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:19:28Z Disabled control of system clock
I20240502 08:19:28.583518  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:41659
--webserver_interface=127.6.151.126
--webserver_port=0
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.151.126:41659
--location_mapping_cmd=/tmp/dist-test-taskOUdoRy/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 08:19:28.922919 24615 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:19:28.923259 24615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:28.923377 24615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:28.923466 24615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:28.931252 24615 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:28.931376 24615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:19:28.931488 24615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:28.931548 24615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:19:28.931636 24615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:19:28.941134 24615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskOUdoRy/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.6.151.126:41659
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.151.126:41659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.151.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true

Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:28.942490 24615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:28.944263 24615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:28.955523 24620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:28.955687 24621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:28.956346 24623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:28.957175 24615 server_base.cc:1034] running on GCE node
I20240502 08:19:28.958084 24615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:28.959586 24615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:28.960870 24615 hybrid_clock.cc:648] HybridClock initialized: now 1714637968960803 us; error 40 us; skew 500 ppm
I20240502 08:19:28.961246 24615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:28.963558 24615 webserver.cc:460] Webserver started at http://127.6.151.126:44897/ using document root <none> and password file <none>
I20240502 08:19:28.964300 24615 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:28.964433 24615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:28.964733 24615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:28.967404 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0ab840839aab43378dcd2b92f6edf949"
format_stamp: "Formatted at 2024-05-02 08:19:28 on dist-test-slave-x3qn"
I20240502 08:19:28.968186 24615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0ab840839aab43378dcd2b92f6edf949"
format_stamp: "Formatted at 2024-05-02 08:19:28 on dist-test-slave-x3qn"
I20240502 08:19:28.973376 24615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20240502 08:19:28.976712 24629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:28.978044 24615 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:28.978214 24615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0ab840839aab43378dcd2b92f6edf949"
format_stamp: "Formatted at 2024-05-02 08:19:28 on dist-test-slave-x3qn"
I20240502 08:19:28.978394 24615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:29.019758 24615 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:29.020658 24615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:29.020984 24615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:29.039932 24615 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.126:41659
I20240502 08:19:29.039970 24680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.126:41659 every 8 connection(s)
I20240502 08:19:29.041482 24615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:19:29.043013  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 24615
I20240502 08:19:29.043237  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:19:29.046519 24681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:29.058272 24681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949: Bootstrap starting.
I20240502 08:19:29.061344 24681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:29.062419 24681 log.cc:824] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:29.065030 24681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949: No bootstrap required, opened a new log
I20240502 08:19:29.071451 24681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab840839aab43378dcd2b92f6edf949" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 41659 } }
I20240502 08:19:29.071846 24681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:29.071951 24681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab840839aab43378dcd2b92f6edf949, State: Initialized, Role: FOLLOWER
I20240502 08:19:29.072796 24681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab840839aab43378dcd2b92f6edf949" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 41659 } }
I20240502 08:19:29.073019 24681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:19:29.073148 24681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:19:29.073299 24681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:29.075708 24681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab840839aab43378dcd2b92f6edf949" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 41659 } }
I20240502 08:19:29.076185 24681 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab840839aab43378dcd2b92f6edf949; no voters: 
I20240502 08:19:29.076756 24681 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20240502 08:19:29.076901 24686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:29.077543 24686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [term 1 LEADER]: Becoming Leader. State: Replica: 0ab840839aab43378dcd2b92f6edf949, State: Running, Role: LEADER
I20240502 08:19:29.078092 24686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab840839aab43378dcd2b92f6edf949" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 41659 } }
I20240502 08:19:29.078670 24681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:19:29.081130 24687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab840839aab43378dcd2b92f6edf949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab840839aab43378dcd2b92f6edf949" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 41659 } } }
I20240502 08:19:29.081568 24687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [sys.catalog]: This master's current role is: LEADER
I20240502 08:19:29.081739 24688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab840839aab43378dcd2b92f6edf949. Latest consensus state: current_term: 1 leader_uuid: "0ab840839aab43378dcd2b92f6edf949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab840839aab43378dcd2b92f6edf949" member_type: VOTER last_known_addr { host: "127.6.151.126" port: 41659 } } }
I20240502 08:19:29.082048 24688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949 [sys.catalog]: This master's current role is: LEADER
I20240502 08:19:29.084152 24695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:19:29.089354 24695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:19:29.097021 24695 catalog_manager.cc:1347] Generated new cluster ID: 5df03ddd742845378da75b5e24f0fc75
I20240502 08:19:29.097151 24695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:19:29.109541 24695 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:19:29.111364 24695 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:19:29.124732 24695 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0ab840839aab43378dcd2b92f6edf949: Generated new TSK 0
I20240502 08:19:29.125880 24695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:19:29.139761  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.65:0
--local_ip_for_outbound_sockets=127.6.151.65
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:19:29.492704 24705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:29.493016 24705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:29.493136 24705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:29.500779 24705 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:29.500921 24705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:29.501088 24705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.65
I20240502 08:19:29.510330 24705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.151.65
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.65
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:29.511701 24705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:29.513415 24705 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:29.525444 24713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:29.526085 24711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:29.526312 24710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:29.527658 24705 server_base.cc:1034] running on GCE node
I20240502 08:19:29.528277 24705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:29.529203 24705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:29.530417 24705 hybrid_clock.cc:648] HybridClock initialized: now 1714637969530369 us; error 34 us; skew 500 ppm
I20240502 08:19:29.530782 24705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:29.543960 24705 webserver.cc:460] Webserver started at http://127.6.151.65:38431/ using document root <none> and password file <none>
I20240502 08:19:29.544790 24705 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:29.544926 24705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:29.545243 24705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:29.548036 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3ccdae65238f49c7aa130138a4dec610"
format_stamp: "Formatted at 2024-05-02 08:19:29 on dist-test-slave-x3qn"
I20240502 08:19:29.548841 24705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3ccdae65238f49c7aa130138a4dec610"
format_stamp: "Formatted at 2024-05-02 08:19:29 on dist-test-slave-x3qn"
I20240502 08:19:29.554170 24705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.003s
I20240502 08:19:29.557940 24719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:29.559410 24705 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:19:29.559579 24705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3ccdae65238f49c7aa130138a4dec610"
format_stamp: "Formatted at 2024-05-02 08:19:29 on dist-test-slave-x3qn"
I20240502 08:19:29.559793 24705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:29.586407 24705 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:29.587312 24705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:29.587585 24705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:29.588546 24705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:29.590003 24705 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:29.590098 24705 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:29.590210 24705 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:29.590281 24705 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:29.622723 24705 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.65:34967
I20240502 08:19:29.622810 24831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.65:34967 every 8 connection(s)
I20240502 08:19:29.624120 24705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:19:29.627805  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 24705
I20240502 08:19:29.628036  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:19:29.631840  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.66:0
--local_ip_for_outbound_sockets=127.6.151.66
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:29.639113 24832 heartbeater.cc:347] Connected to a master server at 127.6.151.126:41659
I20240502 08:19:29.639436 24832 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:29.640256 24832 heartbeater.cc:511] Master 127.6.151.126:41659 requested a full tablet report, sending...
I20240502 08:19:29.688306 24646 ts_manager.cc:194] Registered new tserver with Master: 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:29.690848 24646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.65:43603
W20240502 08:19:29.975858 24836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:29.976168 24836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:29.976258 24836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:29.984127 24836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:29.984282 24836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:29.984413 24836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.66
I20240502 08:19:29.993613 24836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.151.66
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.66
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:29.995081 24836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:29.996879 24836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:30.009203 24842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:30.009536 24843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:30.009799 24845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:30.010546 24836 server_base.cc:1034] running on GCE node
I20240502 08:19:30.011206 24836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:30.012197 24836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:30.013428 24836 hybrid_clock.cc:648] HybridClock initialized: now 1714637970013365 us; error 59 us; skew 500 ppm
I20240502 08:19:30.013769 24836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:30.016224 24836 webserver.cc:460] Webserver started at http://127.6.151.66:38301/ using document root <none> and password file <none>
I20240502 08:19:30.017021 24836 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:30.017146 24836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:30.017474 24836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:30.020280 24836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.021112 24836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.026798 24836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.001s
I20240502 08:19:30.030478 24851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:30.032065 24836 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:19:30.032248 24836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.032470 24836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:30.055859 24836 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:30.056926 24836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:30.057286 24836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:30.058609 24836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:30.060598 24836 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:30.060710 24836 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:30.060822 24836 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:30.060892 24836 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:30.095111 24836 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.66:35329
I20240502 08:19:30.095165 24963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.66:35329 every 8 connection(s)
I20240502 08:19:30.096444 24836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:19:30.097292  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 24836
I20240502 08:19:30.097512  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:19:30.100930  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.67:0
--local_ip_for_outbound_sockets=127.6.151.67
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:30.109831 24964 heartbeater.cc:347] Connected to a master server at 127.6.151.126:41659
I20240502 08:19:30.110142 24964 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:30.111029 24964 heartbeater.cc:511] Master 127.6.151.126:41659 requested a full tablet report, sending...
I20240502 08:19:30.161530 24646 ts_manager.cc:194] Registered new tserver with Master: e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:30.162901 24646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.66:55659
W20240502 08:19:30.466177 24967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:30.466478 24967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:30.466569 24967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:30.474202 24967 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:30.474351 24967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:30.474468 24967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.67
I20240502 08:19:30.484052 24967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.151.67
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.67
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:30.485440 24967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:30.487370 24967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:30.499497 24976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:30.500213 24974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:30.500636 24973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:30.501128 24967 server_base.cc:1034] running on GCE node
I20240502 08:19:30.501758 24967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:30.502764 24967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:30.503973 24967 hybrid_clock.cc:648] HybridClock initialized: now 1714637970503922 us; error 46 us; skew 500 ppm
I20240502 08:19:30.504293 24967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:30.506604 24967 webserver.cc:460] Webserver started at http://127.6.151.67:39417/ using document root <none> and password file <none>
I20240502 08:19:30.507400 24967 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:30.507529 24967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:30.507841 24967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:30.510615 24967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.511494 24967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.517158 24967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20240502 08:19:30.520761 24982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:30.522187 24967 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:30.522353 24967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.522585 24967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:30.540786 24967 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:30.541657 24967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:30.541931 24967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:30.542948 24967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:30.544410 24967 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:30.544517 24967 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:30.544632 24967 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:30.544710 24967 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:30.577176 24967 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.67:45829
I20240502 08:19:30.577219 25094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.67:45829 every 8 connection(s)
I20240502 08:19:30.578524 24967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:19:30.587740  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 24967
I20240502 08:19:30.587900  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:19:30.591190  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:0
--local_ip_for_outbound_sockets=127.6.151.68
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:30.591701 25095 heartbeater.cc:347] Connected to a master server at 127.6.151.126:41659
I20240502 08:19:30.591996 25095 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:30.592826 25095 heartbeater.cc:511] Master 127.6.151.126:41659 requested a full tablet report, sending...
I20240502 08:19:30.638815 24646 ts_manager.cc:194] Registered new tserver with Master: b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:30.639763 24646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.67:47133
I20240502 08:19:30.694044 24832 heartbeater.cc:503] Master 127.6.151.126:41659 was elected leader, sending a full tablet report...
W20240502 08:19:30.945099 25099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:30.945428 25099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:30.945560 25099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:30.953267 25099 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:30.953435 25099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:30.953608 25099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:19:30.963135 25099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:30.964510 25099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:30.966302 25099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:30.978099 25107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:30.978335 25104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:30.978951 25105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:30.979732 25099 server_base.cc:1034] running on GCE node
I20240502 08:19:30.980379 25099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:30.981317 25099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:30.982520 25099 hybrid_clock.cc:648] HybridClock initialized: now 1714637970982476 us; error 36 us; skew 500 ppm
I20240502 08:19:30.982863 25099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:30.985291 25099 webserver.cc:460] Webserver started at http://127.6.151.68:36121/ using document root <none> and password file <none>
I20240502 08:19:30.986052 25099 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:30.986168 25099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:30.986461 25099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:30.989075 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "40fa9dc1519b476d813eb3f71e25d73c"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.989867 25099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "40fa9dc1519b476d813eb3f71e25d73c"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:30.995075 25099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20240502 08:19:30.998467 25113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:30.999804 25099 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20240502 08:19:30.999970 25099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "40fa9dc1519b476d813eb3f71e25d73c"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:31.000169 25099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:31.030316 25099 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:31.031345 25099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:31.031613 25099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:31.032639 25099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:31.034178 25099 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:31.034274 25099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:31.034397 25099 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:31.034477 25099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:31.067476 25099 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:43407
I20240502 08:19:31.067535 25225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:43407 every 8 connection(s)
I20240502 08:19:31.068804 25099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:19:31.076995  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 25099
I20240502 08:19:31.077183  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:19:31.081471  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:0
--local_ip_for_outbound_sockets=127.6.151.69
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:31.083568 25226 heartbeater.cc:347] Connected to a master server at 127.6.151.126:41659
I20240502 08:19:31.083941 25226 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:31.084836 25226 heartbeater.cc:511] Master 127.6.151.126:41659 requested a full tablet report, sending...
I20240502 08:19:31.155941 24646 ts_manager.cc:194] Registered new tserver with Master: 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407)
I20240502 08:19:31.156967 24646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:46617
I20240502 08:19:31.166709 24964 heartbeater.cc:503] Master 127.6.151.126:41659 was elected leader, sending a full tablet report...
W20240502 08:19:31.462630 25230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:31.463011 25230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:31.463119 25230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:31.471557 25230 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:31.471748 25230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:31.471921 25230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:19:31.481453 25230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=0
--tserver_master_addrs=127.6.151.126:41659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:31.482962 25230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:31.484907 25230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:31.497615 25236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:31.497728 25235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:31.498361 25238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:31.499078 25230 server_base.cc:1034] running on GCE node
I20240502 08:19:31.499681 25230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:31.500669 25230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:31.501875 25230 hybrid_clock.cc:648] HybridClock initialized: now 1714637971501829 us; error 37 us; skew 500 ppm
I20240502 08:19:31.502218 25230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:31.504786 25230 webserver.cc:460] Webserver started at http://127.6.151.69:46837/ using document root <none> and password file <none>
I20240502 08:19:31.505590 25230 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:31.505702 25230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:31.506047 25230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:19:31.508869 25230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "c809bcf4be094d1b9a931b8431fdc60c"
format_stamp: "Formatted at 2024-05-02 08:19:31 on dist-test-slave-x3qn"
I20240502 08:19:31.509732 25230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "c809bcf4be094d1b9a931b8431fdc60c"
format_stamp: "Formatted at 2024-05-02 08:19:31 on dist-test-slave-x3qn"
I20240502 08:19:31.515493 25230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20240502 08:19:31.519265 25244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:31.520715 25230 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.003s
I20240502 08:19:31.520897 25230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c809bcf4be094d1b9a931b8431fdc60c"
format_stamp: "Formatted at 2024-05-02 08:19:31 on dist-test-slave-x3qn"
I20240502 08:19:31.521122 25230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:31.539880 25230 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:31.540840 25230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:31.541132 25230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:31.542204 25230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:31.543805 25230 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:31.543910 25230 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:31.544032 25230 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:31.544114 25230 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:31.576889 25230 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:36255
I20240502 08:19:31.576982 25356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:36255 every 8 connection(s)
I20240502 08:19:31.578246 25230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:19:31.584151  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 25230
I20240502 08:19:31.584365  6749 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:19:31.591782 25357 heartbeater.cc:347] Connected to a master server at 127.6.151.126:41659
I20240502 08:19:31.592106 25357 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:31.592934 25357 heartbeater.cc:511] Master 127.6.151.126:41659 requested a full tablet report, sending...
I20240502 08:19:31.640619 24646 ts_manager.cc:194] Registered new tserver with Master: c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255)
I20240502 08:19:31.641875  6749 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:19:31.641913 24646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:41907
I20240502 08:19:31.642488 25095 heartbeater.cc:503] Master 127.6.151.126:41659 was elected leader, sending a full tablet report...
I20240502 08:19:31.659237  6749 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:19:31.659379  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 25230
I20240502 08:19:31.668273 25352 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:31.751942  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 25230
I20240502 08:19:31.768916  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 25099
I20240502 08:19:31.777232 25221 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:31.862349  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 25099
I20240502 08:19:35.631683 24646 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57774:
name: "rebalance_test_table_0"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 08:19:35.638833 24646 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:19:35.672706 24766 tablet_service.cc:1467] Processing CreateTablet for tablet 614c1e62e62e401baad4df6d5270ab57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.672897 24899 tablet_service.cc:1467] Processing CreateTablet for tablet 037baac9651a44cfba46345bba7904cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.673796 24767 tablet_service.cc:1467] Processing CreateTablet for tablet 037baac9651a44cfba46345bba7904cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.674307 24899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037baac9651a44cfba46345bba7904cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.674131 25029 tablet_service.cc:1467] Processing CreateTablet for tablet 614c1e62e62e401baad4df6d5270ab57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.674110 25030 tablet_service.cc:1467] Processing CreateTablet for tablet 037baac9651a44cfba46345bba7904cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.675092 24767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037baac9651a44cfba46345bba7904cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.675292 25028 tablet_service.cc:1467] Processing CreateTablet for tablet a4efbd31bd574cb1902d6856c4da2171 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.675683 25029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c1e62e62e401baad4df6d5270ab57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.676100 25028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efbd31bd574cb1902d6856c4da2171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.677138 25030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037baac9651a44cfba46345bba7904cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.672724 24898 tablet_service.cc:1467] Processing CreateTablet for tablet 614c1e62e62e401baad4df6d5270ab57 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.679073 24898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c1e62e62e401baad4df6d5270ab57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.674136 24765 tablet_service.cc:1467] Processing CreateTablet for tablet a4efbd31bd574cb1902d6856c4da2171 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.679497 24765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efbd31bd574cb1902d6856c4da2171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.679847 24897 tablet_service.cc:1467] Processing CreateTablet for tablet a4efbd31bd574cb1902d6856c4da2171 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:35.680294 24897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efbd31bd574cb1902d6856c4da2171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.680514 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c1e62e62e401baad4df6d5270ab57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:35.702988 25383 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:35.703600 25381 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:35.704007 25382 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:35.706841 25383 tablet_bootstrap.cc:654] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.707217 25381 tablet_bootstrap.cc:654] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.707664 25382 tablet_bootstrap.cc:654] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.708233 25383 log.cc:824] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:35.708532 25381 log.cc:824] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:35.708791 25382 log.cc:824] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:35.716312 25382 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:35.716324 25381 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:35.716318 25383 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:35.716586 25382 ts_tablet_manager.cc:1367] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20240502 08:19:35.716599 25383 ts_tablet_manager.cc:1367] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.004s
I20240502 08:19:35.716593 25381 ts_tablet_manager.cc:1367] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.013s	user 0.005s	sys 0.006s
I20240502 08:19:35.723192 25382 raft_consensus.cc:348] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.723562 25382 raft_consensus.cc:374] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.723677 25382 raft_consensus.cc:729] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.724329 25382 consensus_queue.cc:260] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.725523 25381 raft_consensus.cc:348] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.725525 25383 raft_consensus.cc:348] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.725939 25381 raft_consensus.cc:374] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.725939 25383 raft_consensus.cc:374] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.726045 25383 raft_consensus.cc:729] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.726122 25381 raft_consensus.cc:729] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.726879 25383 consensus_queue.cc:260] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.727416 25382 ts_tablet_manager.cc:1398] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.011s	user 0.012s	sys 0.000s
I20240502 08:19:35.727574 25381 consensus_queue.cc:260] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.727919 25382 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:35.730259 25382 tablet_bootstrap.cc:654] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.738042 25382 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:35.738279 25382 ts_tablet_manager.cc:1367] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.010s	user 0.000s	sys 0.009s
I20240502 08:19:35.738384 25381 ts_tablet_manager.cc:1398] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.022s	user 0.016s	sys 0.004s
I20240502 08:19:35.738941 25381 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:35.739199 25383 ts_tablet_manager.cc:1398] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.023s	user 0.016s	sys 0.006s
I20240502 08:19:35.739396 25382 raft_consensus.cc:348] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.739656 25382 raft_consensus.cc:374] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.739756 25383 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:35.739794 25382 raft_consensus.cc:729] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.740114 25382 consensus_queue.cc:260] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.740708 25381 tablet_bootstrap.cc:654] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.740705 25382 ts_tablet_manager.cc:1398] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:35.740959 25382 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:35.742576 25382 tablet_bootstrap.cc:654] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.744984 25381 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:35.745211 25381 ts_tablet_manager.cc:1367] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240502 08:19:35.745380 25383 tablet_bootstrap.cc:654] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.746026 25381 raft_consensus.cc:348] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.746145 25382 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:35.746217 25381 raft_consensus.cc:374] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.746292 25382 ts_tablet_manager.cc:1367] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:19:35.746323 25381 raft_consensus.cc:729] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.746618 25381 consensus_queue.cc:260] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.747017 25382 raft_consensus.cc:348] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.747253 25381 ts_tablet_manager.cc:1398] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.001s
I20240502 08:19:35.747298 25382 raft_consensus.cc:374] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.747434 25382 raft_consensus.cc:729] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.747629 25381 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:35.747731 25382 consensus_queue.cc:260] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.747936 25383 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:35.748112 25383 ts_tablet_manager.cc:1367] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.009s	user 0.003s	sys 0.001s
I20240502 08:19:35.749187 25382 ts_tablet_manager.cc:1398] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.000s
I20240502 08:19:35.749177 25389 raft_consensus.cc:482] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:35.749435 25389 raft_consensus.cc:504] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.749752 25383 raft_consensus.cc:348] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.749948 25383 raft_consensus.cc:374] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.750492 25383 raft_consensus.cc:729] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.750814 25383 consensus_queue.cc:260] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.751598 25389 leader_election.cc:288] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:35.751715 25383 ts_tablet_manager.cc:1398] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.003s	user 0.001s	sys 0.001s
I20240502 08:19:35.751981 25383 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:35.753715 25383 tablet_bootstrap.cc:654] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.749455 25381 tablet_bootstrap.cc:654] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:35.761814 25383 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:35.762020 25383 ts_tablet_manager.cc:1367] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.010s	user 0.004s	sys 0.005s
I20240502 08:19:35.762863 25383 raft_consensus.cc:348] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.763154 25383 raft_consensus.cc:374] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.763223 25383 raft_consensus.cc:729] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.763463 25383 consensus_queue.cc:260] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.764096 25383 ts_tablet_manager.cc:1398] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:35.766973 25381 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:35.767186 25381 ts_tablet_manager.cc:1367] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.002s
I20240502 08:19:35.768013 25381 raft_consensus.cc:348] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.768327 25381 raft_consensus.cc:374] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:35.768450 25381 raft_consensus.cc:729] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:35.768730 25381 consensus_queue.cc:260] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.769377 25381 ts_tablet_manager.cc:1398] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:19:35.769538 24787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037baac9651a44cfba46345bba7904cc" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610" is_pre_election: true
I20240502 08:19:35.770099 24787 raft_consensus.cc:2454] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 0.
I20240502 08:19:35.770972 24983 leader_election.cc:302] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, b4fd6df034eb4418ab29ad24a7a8a9fd; no voters: 
I20240502 08:19:35.771402 25389 raft_consensus.cc:2781] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:35.771561 25389 raft_consensus.cc:482] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:35.771667 25389 raft_consensus.cc:3037] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.773649 24919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037baac9651a44cfba46345bba7904cc" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" is_pre_election: true
I20240502 08:19:35.774130 24919 raft_consensus.cc:2454] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 0.
I20240502 08:19:35.774142 25389 raft_consensus.cc:504] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.774757 25389 leader_election.cc:288] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 election: Requested vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:35.775110 24787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037baac9651a44cfba46345bba7904cc" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610"
I20240502 08:19:35.775275 24787 raft_consensus.cc:3037] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.775288 24919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037baac9651a44cfba46345bba7904cc" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:35.775456 24919 raft_consensus.cc:3037] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.777348 24787 raft_consensus.cc:2454] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 1.
I20240502 08:19:35.777442 24919 raft_consensus.cc:2454] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 1.
I20240502 08:19:35.777767 24983 leader_election.cc:302] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, b4fd6df034eb4418ab29ad24a7a8a9fd; no voters: 
I20240502 08:19:35.778111 25389 raft_consensus.cc:2781] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:35.778620 25389 raft_consensus.cc:686] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Becoming Leader. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Running, Role: LEADER
I20240502 08:19:35.779162 25389 consensus_queue.cc:237] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.784330 24644 catalog_manager.cc:5484] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: term changed from 0 to 1, leader changed from <none> to b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67). New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:35.785487 25387 raft_consensus.cc:482] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:35.785646 25387 raft_consensus.cc:504] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.787272 25387 leader_election.cc:288] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:35.794507 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c1e62e62e401baad4df6d5270ab57" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" is_pre_election: true
I20240502 08:19:35.795017 25050 raft_consensus.cc:2454] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 0.
I20240502 08:19:35.795579 24853 leader_election.cc:302] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4fd6df034eb4418ab29ad24a7a8a9fd, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:35.795998 25387 raft_consensus.cc:2781] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:35.796131 25387 raft_consensus.cc:482] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:35.796223 25387 raft_consensus.cc:3037] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.796448 24787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c1e62e62e401baad4df6d5270ab57" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610" is_pre_election: true
I20240502 08:19:35.796669 24787 raft_consensus.cc:2454] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 0.
I20240502 08:19:35.798409 25387 raft_consensus.cc:504] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.799044 25387 leader_election.cc:288] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 election: Requested vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:35.799422 24787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c1e62e62e401baad4df6d5270ab57" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610"
I20240502 08:19:35.799533 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "614c1e62e62e401baad4df6d5270ab57" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:35.799614 24787 raft_consensus.cc:3037] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.799716 25050 raft_consensus.cc:3037] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.801448 24787 raft_consensus.cc:2454] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 1.
I20240502 08:19:35.801687 25050 raft_consensus.cc:2454] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 1.
I20240502 08:19:35.801820 24852 leader_election.cc:302] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:35.802119 25387 raft_consensus.cc:2781] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:35.802486 25387 raft_consensus.cc:686] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Becoming Leader. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Running, Role: LEADER
I20240502 08:19:35.803035 25387 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.807461 24644 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: UNKNOWN } } }
W20240502 08:19:35.833901 25096 tablet.cc:2369] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:19:35.852777 24965 tablet.cc:2369] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:19:35.881008 24833 tablet.cc:2369] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:19:35.918825 25388 raft_consensus.cc:482] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:35.919050 25388 raft_consensus.cc:504] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.920661 25388 leader_election.cc:288] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:35.928617 24919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efbd31bd574cb1902d6856c4da2171" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" is_pre_election: true
I20240502 08:19:35.928893 24919 raft_consensus.cc:2454] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:35.928946 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efbd31bd574cb1902d6856c4da2171" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" is_pre_election: true
I20240502 08:19:35.929216 25050 raft_consensus.cc:2454] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:35.929404 24721 leader_election.cc:302] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:35.929867 25388 raft_consensus.cc:2781] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:35.929987 25388 raft_consensus.cc:482] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:35.930076 25388 raft_consensus.cc:3037] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.932202 25388 raft_consensus.cc:504] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.932840 25388 leader_election.cc:288] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Requested vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:35.933373 24919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efbd31bd574cb1902d6856c4da2171" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:35.933444 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efbd31bd574cb1902d6856c4da2171" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:35.933606 24919 raft_consensus.cc:3037] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.933665 25050 raft_consensus.cc:3037] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:35.935663 25050 raft_consensus.cc:2454] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:35.935676 24919 raft_consensus.cc:2454] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:35.936095 24721 leader_election.cc:302] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:35.936414 25388 raft_consensus.cc:2781] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:35.936852 25388 raft_consensus.cc:686] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Running, Role: LEADER
I20240502 08:19:35.937434 25388 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:35.941733 24644 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:36.279170 25389 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:36.289578 25389 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:36.324190 25408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:36.324524 25408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:36.333719 25408 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 08:19:36.345059 25388 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:36.349620 25388 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 08:19:36.405094 25387 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:36.418298 25441 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 281.252 ms
  time per row: 0.140626 ms
-----------------------------------------------------
Suppressions used:
  count      bytes template
      2        596 libcrypto.so.1.1
-----------------------------------------------------

I20240502 08:19:37.713604 24641 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57774:
name: "rebalance_test_table_1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 08:19:37.714367 24641 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:19:37.722996 24765 tablet_service.cc:1467] Processing CreateTablet for tablet 17d1d1592ecc4d52addd77be44075a1b (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.723140 24897 tablet_service.cc:1467] Processing CreateTablet for tablet 17d1d1592ecc4d52addd77be44075a1b (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.723429 24765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d1d1592ecc4d52addd77be44075a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.723600 24897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d1d1592ecc4d52addd77be44075a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.724705 24767 tablet_service.cc:1467] Processing CreateTablet for tablet cf7e142110e04e9db0fdc2c5138117ae (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.725096 24767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7e142110e04e9db0fdc2c5138117ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.727905 24766 tablet_service.cc:1467] Processing CreateTablet for tablet 75df7ed7df824d68a02d8b8ebb92db13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.728291 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df7ed7df824d68a02d8b8ebb92db13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.733440 25028 tablet_service.cc:1467] Processing CreateTablet for tablet 75df7ed7df824d68a02d8b8ebb92db13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.734076 25028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df7ed7df824d68a02d8b8ebb92db13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.733964 25030 tablet_service.cc:1467] Processing CreateTablet for tablet 17d1d1592ecc4d52addd77be44075a1b (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.734478 25030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d1d1592ecc4d52addd77be44075a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.735241 25480 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:37.735637 25479 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:37.737079 25029 tablet_service.cc:1467] Processing CreateTablet for tablet cf7e142110e04e9db0fdc2c5138117ae (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.737454 25479 tablet_bootstrap.cc:654] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.737493 25029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7e142110e04e9db0fdc2c5138117ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.737794 25480 tablet_bootstrap.cc:654] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.743281 24899 tablet_service.cc:1467] Processing CreateTablet for tablet 75df7ed7df824d68a02d8b8ebb92db13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.743878 24899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df7ed7df824d68a02d8b8ebb92db13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.746271 25480 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:37.746505 25480 ts_tablet_manager.cc:1367] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.011s	user 0.005s	sys 0.005s
I20240502 08:19:37.747139 24898 tablet_service.cc:1467] Processing CreateTablet for tablet cf7e142110e04e9db0fdc2c5138117ae (DEFAULT_TABLE table=rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:37.747586 24898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7e142110e04e9db0fdc2c5138117ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:37.747377 25480 raft_consensus.cc:348] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.747668 25480 raft_consensus.cc:374] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.747756 25480 raft_consensus.cc:729] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.748054 25480 consensus_queue.cc:260] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.748549 25483 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:37.749346 25480 ts_tablet_manager.cc:1398] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.002s
I20240502 08:19:37.750878 25483 tablet_bootstrap.cc:654] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.759967 25483 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:37.760361 25483 ts_tablet_manager.cc:1367] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.006s
I20240502 08:19:37.761547 25483 raft_consensus.cc:348] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.761960 25483 raft_consensus.cc:374] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.762137 25483 raft_consensus.cc:729] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.762563 25483 consensus_queue.cc:260] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.763438 25483 ts_tablet_manager.cc:1398] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:37.764523 25480 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:37.766707 25480 tablet_bootstrap.cc:654] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.770483 25483 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:37.772686 25483 tablet_bootstrap.cc:654] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.775156 25483 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:37.775298 25479 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:37.775352 25483 ts_tablet_manager.cc:1367] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.005s	user 0.004s	sys 0.000s
I20240502 08:19:37.775434 25480 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:37.775533 25479 ts_tablet_manager.cc:1367] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.040s	user 0.027s	sys 0.013s
I20240502 08:19:37.775617 25480 ts_tablet_manager.cc:1367] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.011s	user 0.000s	sys 0.009s
I20240502 08:19:37.776068 25483 raft_consensus.cc:348] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.776258 25483 raft_consensus.cc:374] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.776337 25483 raft_consensus.cc:729] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.776340 25480 raft_consensus.cc:348] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.776309 25479 raft_consensus.cc:348] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.776541 25479 raft_consensus.cc:374] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.776561 25480 raft_consensus.cc:374] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.776633 25480 raft_consensus.cc:729] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.776679 25479 raft_consensus.cc:729] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.776772 25483 consensus_queue.cc:260] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.776924 25479 consensus_queue.cc:260] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.776942 25480 consensus_queue.cc:260] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.777518 25479 ts_tablet_manager.cc:1398] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:37.778122 25483 ts_tablet_manager.cc:1398] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:37.778461 25483 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:37.778910 25480 ts_tablet_manager.cc:1398] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.003s	user 0.000s	sys 0.003s
I20240502 08:19:37.779326 25480 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:37.779352 25479 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:37.781035 25479 tablet_bootstrap.cc:654] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.781724 25480 tablet_bootstrap.cc:654] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.781914 25483 tablet_bootstrap.cc:654] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.787158 25480 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:37.787338 25480 ts_tablet_manager.cc:1367] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.008s	user 0.004s	sys 0.002s
I20240502 08:19:37.787545 25479 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:37.787726 25479 ts_tablet_manager.cc:1367] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.010s	user 0.002s	sys 0.005s
I20240502 08:19:37.788209 25480 raft_consensus.cc:348] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.788434 25480 raft_consensus.cc:374] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.788465 25479 raft_consensus.cc:348] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.788589 25480 raft_consensus.cc:729] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.788707 25479 raft_consensus.cc:374] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.788790 25479 raft_consensus.cc:729] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.788844 25480 consensus_queue.cc:260] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.789029 25479 consensus_queue.cc:260] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.789469 25480 ts_tablet_manager.cc:1398] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:19:37.789710 25479 ts_tablet_manager.cc:1398] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:19:37.790035 25479 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:37.791682 25479 tablet_bootstrap.cc:654] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:37.793555 25483 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:37.793836 25483 ts_tablet_manager.cc:1367] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.015s	user 0.006s	sys 0.002s
I20240502 08:19:37.794721 25483 raft_consensus.cc:348] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.794951 25483 raft_consensus.cc:374] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.795061 25483 raft_consensus.cc:729] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.795405 25479 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:37.795301 25483 consensus_queue.cc:260] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.795545 25479 ts_tablet_manager.cc:1367] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.006s	user 0.000s	sys 0.004s
I20240502 08:19:37.795874 25483 ts_tablet_manager.cc:1398] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:19:37.796259 25479 raft_consensus.cc:348] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.796444 25479 raft_consensus.cc:374] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:37.796545 25479 raft_consensus.cc:729] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:37.796773 25479 consensus_queue.cc:260] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.797343 25479 ts_tablet_manager.cc:1398] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:19:37.905483 25474 raft_consensus.cc:482] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:37.905692 25474 raft_consensus.cc:504] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.906347 25474 leader_election.cc:288] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:37.906421 25406 raft_consensus.cc:482] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:37.906633 25406 raft_consensus.cc:504] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.907013 25049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7e142110e04e9db0fdc2c5138117ae" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" is_pre_election: true
I20240502 08:19:37.907019 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7e142110e04e9db0fdc2c5138117ae" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610" is_pre_election: true
I20240502 08:19:37.907253 24786 raft_consensus.cc:2454] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 0.
I20240502 08:19:37.907248 25049 raft_consensus.cc:2454] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 0.
I20240502 08:19:37.907450 25406 leader_election.cc:288] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:37.907747 24852 leader_election.cc:302] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:37.908104 25049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df7ed7df824d68a02d8b8ebb92db13" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" is_pre_election: true
I20240502 08:19:37.908175 25474 raft_consensus.cc:2781] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:37.908303 25474 raft_consensus.cc:482] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:37.908301 25049 raft_consensus.cc:2454] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:37.908406 25474 raft_consensus.cc:3037] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:37.908504 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df7ed7df824d68a02d8b8ebb92db13" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" is_pre_election: true
I20240502 08:19:37.908722 24918 raft_consensus.cc:2454] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:37.908718 24721 leader_election.cc:302] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, b4fd6df034eb4418ab29ad24a7a8a9fd; no voters: 
I20240502 08:19:37.908988 25406 raft_consensus.cc:2781] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:37.909106 25406 raft_consensus.cc:482] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:37.909193 25406 raft_consensus.cc:3037] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:37.911327 25474 raft_consensus.cc:504] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.912057 25474 leader_election.cc:288] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 election: Requested vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:37.911947 25406 raft_consensus.cc:504] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.912503 25049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7e142110e04e9db0fdc2c5138117ae" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:37.912525 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf7e142110e04e9db0fdc2c5138117ae" candidate_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610"
I20240502 08:19:37.912698 24786 raft_consensus.cc:3037] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:37.912714 25049 raft_consensus.cc:3037] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:37.913070 25406 leader_election.cc:288] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Requested vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:37.913583 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df7ed7df824d68a02d8b8ebb92db13" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:37.913587 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75df7ed7df824d68a02d8b8ebb92db13" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:37.913761 24918 raft_consensus.cc:3037] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:37.913761 25050 raft_consensus.cc:3037] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:37.915493 24786 raft_consensus.cc:2454] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 1.
I20240502 08:19:37.915901 24852 leader_election.cc:302] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:37.916023 25049 raft_consensus.cc:2454] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2cb3e9aae7d4ef7b06b5d1e6501add5 in term 1.
I20240502 08:19:37.916216 25474 raft_consensus.cc:2781] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:37.916523 24918 raft_consensus.cc:2454] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:37.916668 25050 raft_consensus.cc:2454] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:37.916909 25474 raft_consensus.cc:686] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Becoming Leader. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Running, Role: LEADER
I20240502 08:19:37.916920 24721 leader_election.cc:302] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:37.917268 25406 raft_consensus.cc:2781] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:37.917228 25474 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.917634 25406 raft_consensus.cc:686] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Running, Role: LEADER
I20240502 08:19:37.918159 25406 consensus_queue.cc:237] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:37.920424 24637 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66). New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:37.921104 24641 catalog_manager.cc:5484] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:38.108912 25410 raft_consensus.cc:482] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:38.109130 25410 raft_consensus.cc:504] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.109911 25410 leader_election.cc:288] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:38.110471 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610" is_pre_election: true
I20240502 08:19:38.110471 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" is_pre_election: true
I20240502 08:19:38.110736 24918 raft_consensus.cc:2454] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 0.
I20240502 08:19:38.110735 24786 raft_consensus.cc:2454] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 0.
I20240502 08:19:38.111156 24984 leader_election.cc:302] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4fd6df034eb4418ab29ad24a7a8a9fd, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:38.111423 25410 raft_consensus.cc:2781] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:38.111547 25410 raft_consensus.cc:482] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:38.111642 25410 raft_consensus.cc:3037] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:38.113889 25410 raft_consensus.cc:504] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.114460 25410 leader_election.cc:288] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 election: Requested vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:38.114989 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610"
I20240502 08:19:38.115196 24786 raft_consensus.cc:3037] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:38.117059 24786 raft_consensus.cc:2454] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 1.
I20240502 08:19:38.117347 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:38.117555 24918 raft_consensus.cc:3037] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:38.117578 24983 leader_election.cc:302] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, b4fd6df034eb4418ab29ad24a7a8a9fd; no voters: 
I20240502 08:19:38.117916 25410 raft_consensus.cc:2781] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:38.118330 25410 raft_consensus.cc:686] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Becoming Leader. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Running, Role: LEADER
I20240502 08:19:38.119041 25410 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.120316 24918 raft_consensus.cc:2454] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 1.
I20240502 08:19:38.122159 24641 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: term changed from 0 to 1, leader changed from <none> to b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67). New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:38.323623 25474 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:38.329841 25474 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:38.353449 25406 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:38.358350 25492 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:38.557931 25410 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:38.560712 25498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:38.561260 25498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:19:38.563187 25410 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:38.571115 25498 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'

INSERT report
    rows total: 2000
    time total: 226.028 ms
  time per row: 0.113014 ms
I20240502 08:19:38.943105 24641 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57774:
name: "rebalance_test_table_2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
  hash_schema {
    columns {
      name: "key"
    }
    num_buckets: 3
    seed: 0
  }
}
W20240502 08:19:38.943768 24641 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:19:38.951829 24898 tablet_service.cc:1467] Processing CreateTablet for tablet 820325c11c654d54886546151c823403 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.952205 24766 tablet_service.cc:1467] Processing CreateTablet for tablet 820325c11c654d54886546151c823403 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.952355 24898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820325c11c654d54886546151c823403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.952654 24766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820325c11c654d54886546151c823403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.953979 25029 tablet_service.cc:1467] Processing CreateTablet for tablet 820325c11c654d54886546151c823403 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.954068 25030 tablet_service.cc:1467] Processing CreateTablet for tablet 11d0289cfeaa4c30a22c499c92b5b653 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.954433 25029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820325c11c654d54886546151c823403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.954937 24899 tablet_service.cc:1467] Processing CreateTablet for tablet 11d0289cfeaa4c30a22c499c92b5b653 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.955003 24897 tablet_service.cc:1467] Processing CreateTablet for tablet d915caab5c694338a1a2a8ec599e1848 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.955049 25028 tablet_service.cc:1467] Processing CreateTablet for tablet d915caab5c694338a1a2a8ec599e1848 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.955319 24899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d0289cfeaa4c30a22c499c92b5b653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.955384 25028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d915caab5c694338a1a2a8ec599e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.955385 24765 tablet_service.cc:1467] Processing CreateTablet for tablet 11d0289cfeaa4c30a22c499c92b5b653 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.955768 24765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d0289cfeaa4c30a22c499c92b5b653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.957106 24897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d915caab5c694338a1a2a8ec599e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.959192 25030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d0289cfeaa4c30a22c499c92b5b653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.966729 24767 tablet_service.cc:1467] Processing CreateTablet for tablet d915caab5c694338a1a2a8ec599e1848 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:19:38.967393 25564 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:38.969357 25564 tablet_bootstrap.cc:654] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:38.971712 25564 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:38.971915 25564 ts_tablet_manager.cc:1367] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.005s	user 0.003s	sys 0.000s
I20240502 08:19:38.973002 25564 raft_consensus.cc:348] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.973304 25564 raft_consensus.cc:374] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:38.973479 25564 raft_consensus.cc:729] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:38.973886 25564 consensus_queue.cc:260] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.974782 25564 ts_tablet_manager.cc:1398] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.003s	user 0.004s	sys 0.000s
I20240502 08:19:38.975271 25564 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:38.977860 24767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d915caab5c694338a1a2a8ec599e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:38.983619 25563 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:38.984288 25564 tablet_bootstrap.cc:654] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:38.986876 25566 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:38.987444 25563 tablet_bootstrap.cc:654] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:38.988202 25564 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:38.988440 25564 ts_tablet_manager.cc:1367] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.013s	user 0.003s	sys 0.003s
I20240502 08:19:38.989131 25566 tablet_bootstrap.cc:654] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:38.989342 25563 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:38.989488 25563 ts_tablet_manager.cc:1367] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.006s	user 0.004s	sys 0.000s
I20240502 08:19:38.989341 25564 raft_consensus.cc:348] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.989591 25564 raft_consensus.cc:374] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:38.989681 25564 raft_consensus.cc:729] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:38.989974 25564 consensus_queue.cc:260] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.990746 25564 ts_tablet_manager.cc:1398] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:19:38.990226 25563 raft_consensus.cc:348] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.991130 25564 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Bootstrap starting.
I20240502 08:19:38.992350 25563 raft_consensus.cc:374] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:38.992525 25563 raft_consensus.cc:729] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:38.993292 25564 tablet_bootstrap.cc:654] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:38.994218 25563 consensus_queue.cc:260] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.995177 25563 ts_tablet_manager.cc:1398] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20240502 08:19:38.995512 25563 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:38.996286 25566 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:38.996479 25564 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: No bootstrap required, opened a new log
I20240502 08:19:38.996538 25566 ts_tablet_manager.cc:1367] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.010s	user 0.005s	sys 0.003s
I20240502 08:19:38.996665 25564 ts_tablet_manager.cc:1367] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240502 08:19:38.997200 25563 tablet_bootstrap.cc:654] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:38.998003 25564 raft_consensus.cc:348] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.998296 25564 raft_consensus.cc:374] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:38.998464 25564 raft_consensus.cc:729] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Initialized, Role: FOLLOWER
I20240502 08:19:38.998836 25564 consensus_queue.cc:260] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.998881 25566 raft_consensus.cc:348] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:38.999305 25566 raft_consensus.cc:374] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:38.999406 25566 raft_consensus.cc:729] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:38.999657 25564 ts_tablet_manager.cc:1398] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.001s
I20240502 08:19:38.999683 25566 consensus_queue.cc:260] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.000264 25566 ts_tablet_manager.cc:1398] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.004s	user 0.002s	sys 0.000s
I20240502 08:19:39.000654 25566 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:39.001608 25563 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:39.001828 25563 ts_tablet_manager.cc:1367] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.006s	user 0.002s	sys 0.003s
I20240502 08:19:39.002281 25566 tablet_bootstrap.cc:654] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:39.002794 25563 raft_consensus.cc:348] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.003064 25563 raft_consensus.cc:374] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:39.003150 25563 raft_consensus.cc:729] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:39.003485 25563 consensus_queue.cc:260] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.004175 25563 ts_tablet_manager.cc:1398] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:19:39.004534 25563 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Bootstrap starting.
I20240502 08:19:39.004699 25566 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:39.004891 25566 ts_tablet_manager.cc:1367] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.004s	user 0.004s	sys 0.000s
I20240502 08:19:39.005779 25566 raft_consensus.cc:348] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.006011 25566 raft_consensus.cc:374] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:39.006099 25566 raft_consensus.cc:729] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:39.006356 25566 consensus_queue.cc:260] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.006973 25566 ts_tablet_manager.cc:1398] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.000s
I20240502 08:19:39.007084 25563 tablet_bootstrap.cc:654] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:39.007325 25566 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: Bootstrap starting.
I20240502 08:19:39.009085 25566 tablet_bootstrap.cc:654] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: Neither blocks nor log segments found. Creating new log.
I20240502 08:19:39.010466 25563 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5: No bootstrap required, opened a new log
I20240502 08:19:39.010612 25563 ts_tablet_manager.cc:1367] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent bootstrapping tablet: real 0.006s	user 0.001s	sys 0.003s
I20240502 08:19:39.011078 25566 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: No bootstrap required, opened a new log
I20240502 08:19:39.011201 25566 ts_tablet_manager.cc:1367] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: Time spent bootstrapping tablet: real 0.004s	user 0.003s	sys 0.001s
I20240502 08:19:39.011291 25563 raft_consensus.cc:348] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.011457 25563 raft_consensus.cc:374] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:39.011548 25563 raft_consensus.cc:729] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cb3e9aae7d4ef7b06b5d1e6501add5, State: Initialized, Role: FOLLOWER
I20240502 08:19:39.011775 25563 consensus_queue.cc:260] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.011842 25566 raft_consensus.cc:348] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.012250 25563 ts_tablet_manager.cc:1398] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:19:39.012513 25566 raft_consensus.cc:374] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:19:39.012645 25566 raft_consensus.cc:729] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Initialized, Role: FOLLOWER
I20240502 08:19:39.012893 25566 consensus_queue.cc:260] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.013414 25566 ts_tablet_manager.cc:1398] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.002s
I20240502 08:19:39.082816 25406 raft_consensus.cc:482] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:39.083031 25406 raft_consensus.cc:504] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.083781 25406 leader_election.cc:288] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:39.084455 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820325c11c654d54886546151c823403" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" is_pre_election: true
I20240502 08:19:39.084470 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820325c11c654d54886546151c823403" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" is_pre_election: true
I20240502 08:19:39.084686 24918 raft_consensus.cc:2454] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:39.084687 25050 raft_consensus.cc:2454] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:39.085167 24721 leader_election.cc:302] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:39.085462 25406 raft_consensus.cc:2781] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:39.085593 25406 raft_consensus.cc:482] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:39.085697 25406 raft_consensus.cc:3037] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.087639 25406 raft_consensus.cc:504] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.088181 25406 leader_election.cc:288] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Requested vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:39.088651 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820325c11c654d54886546151c823403" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:39.088749 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "820325c11c654d54886546151c823403" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:39.088840 24918 raft_consensus.cc:3037] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.088956 25050 raft_consensus.cc:3037] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.090893 25050 raft_consensus.cc:2454] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:39.090906 24918 raft_consensus.cc:2454] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:39.091296 24721 leader_election.cc:302] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:39.091594 25406 raft_consensus.cc:2781] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:39.091760 25406 raft_consensus.cc:686] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Running, Role: LEADER
I20240502 08:19:39.092060 25406 consensus_queue.cc:237] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.095055 24641 catalog_manager.cc:5484] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:39.099447 25406 raft_consensus.cc:482] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:39.099660 25406 raft_consensus.cc:504] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.100378 25406 leader_election.cc:288] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:39.100940 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d915caab5c694338a1a2a8ec599e1848" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" is_pre_election: true
I20240502 08:19:39.101006 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d915caab5c694338a1a2a8ec599e1848" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" is_pre_election: true
I20240502 08:19:39.101192 24918 raft_consensus.cc:2454] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:39.101212 25050 raft_consensus.cc:2454] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 0.
I20240502 08:19:39.101567 24721 leader_election.cc:302] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:39.101866 25406 raft_consensus.cc:2781] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:39.101963 25406 raft_consensus.cc:482] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:39.102075 25406 raft_consensus.cc:3037] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.104034 25406 raft_consensus.cc:504] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.104540 25406 leader_election.cc:288] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Requested vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:39.105042 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d915caab5c694338a1a2a8ec599e1848" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:39.105314 24918 raft_consensus.cc:3037] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.107573 24918 raft_consensus.cc:2454] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:39.108024 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d915caab5c694338a1a2a8ec599e1848" candidate_uuid: "3ccdae65238f49c7aa130138a4dec610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:39.108223 25050 raft_consensus.cc:3037] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.110160 25050 raft_consensus.cc:2454] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ccdae65238f49c7aa130138a4dec610 in term 1.
I20240502 08:19:39.110507 24721 leader_election.cc:302] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:39.110888 25406 raft_consensus.cc:2781] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:39.111092 25406 raft_consensus.cc:686] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccdae65238f49c7aa130138a4dec610, State: Running, Role: LEADER
I20240502 08:19:39.111374 25406 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.114344 24641 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65). New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:39.125893 25540 raft_consensus.cc:482] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:19:39.126118 25540 raft_consensus.cc:504] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.127002 25540 leader_election.cc:288] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:39.127605 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" is_pre_election: true
I20240502 08:19:39.127616 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610" is_pre_election: true
I20240502 08:19:39.127815 24918 raft_consensus.cc:2454] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 0.
I20240502 08:19:39.127841 24786 raft_consensus.cc:2454] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 0.
I20240502 08:19:39.128369 24984 leader_election.cc:302] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, b4fd6df034eb4418ab29ad24a7a8a9fd, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:39.128717 25540 raft_consensus.cc:2781] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:19:39.128839 25540 raft_consensus.cc:482] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:19:39.128917 25540 raft_consensus.cc:3037] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.131183 25540 raft_consensus.cc:504] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.131794 25540 leader_election.cc:288] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 election: Requested vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:39.132292 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:39.132485 24918 raft_consensus.cc:3037] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.132519 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653" candidate_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccdae65238f49c7aa130138a4dec610"
I20240502 08:19:39.132732 24786 raft_consensus.cc:3037] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:19:39.134408 24918 raft_consensus.cc:2454] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 1.
I20240502 08:19:39.134704 24786 raft_consensus.cc:2454] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4fd6df034eb4418ab29ad24a7a8a9fd in term 1.
I20240502 08:19:39.134886 24984 leader_election.cc:302] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4fd6df034eb4418ab29ad24a7a8a9fd, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:39.135203 25540 raft_consensus.cc:2781] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:19:39.135352 25540 raft_consensus.cc:686] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Becoming Leader. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Running, Role: LEADER
I20240502 08:19:39.135651 25540 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } }
I20240502 08:19:39.138726 24641 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: term changed from 0 to 1, leader changed from <none> to b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67). New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } }
I20240502 08:19:39.520977 25406 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:39.525103 25578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:39.525596 25578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 08:19:39.526758 25492 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:19:39.535738 25578 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:19:39.573482 25492 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:39.578526 25492 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:39.628793 25540 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:19:39.651443 25540 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s

INSERT report
    rows total: 2000
    time total: 213.438 ms
  time per row: 0.106719 ms
I20240502 08:19:39.908634  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.69:36255
--local_ip_for_outbound_sockets=127.6.151.69
--tserver_master_addrs=127.6.151.126:41659
--webserver_port=46837
--webserver_interface=127.6.151.69
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:19:40.274190 25633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:40.274508 25633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:40.274600 25633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:40.282433 25633 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:40.282582 25633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:40.282778 25633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.69
I20240502 08:19:40.292189 25633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.69:36255
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.151.69
--webserver_port=46837
--tserver_master_addrs=127.6.151.126:41659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.69
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:40.293745 25633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:40.295832 25633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:40.308868 25638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:40.309535 25639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:40.309754 25641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:40.310456 25633 server_base.cc:1034] running on GCE node
I20240502 08:19:40.311120 25633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:40.312340 25633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:40.313565 25633 hybrid_clock.cc:648] HybridClock initialized: now 1714637980313508 us; error 47 us; skew 500 ppm
I20240502 08:19:40.313894 25633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:40.316330 25633 webserver.cc:460] Webserver started at http://127.6.151.69:46837/ using document root <none> and password file <none>
I20240502 08:19:40.317097 25633 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:40.317230 25633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:40.322695 25633 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 08:19:40.325505 25647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:40.326856 25633 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20240502 08:19:40.327030 25633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "c809bcf4be094d1b9a931b8431fdc60c"
format_stamp: "Formatted at 2024-05-02 08:19:31 on dist-test-slave-x3qn"
I20240502 08:19:40.327811 25633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:40.348006 25633 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:40.348846 25633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:40.349148 25633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:40.350138 25633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:40.351722 25633 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:40.351804 25633 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:40.351958 25633 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:40.352015 25633 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:40.385191 25633 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.69:36255
I20240502 08:19:40.385228 25759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.69:36255 every 8 connection(s)
I20240502 08:19:40.386673 25633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:19:40.391388  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 25633
I20240502 08:19:40.392050  6749 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
/tmp/dist-test-taskOUdoRy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.151.68:43407
--local_ip_for_outbound_sockets=127.6.151.68
--tserver_master_addrs=127.6.151.126:41659
--webserver_port=36121
--webserver_interface=127.6.151.68
--builtin_ntp_servers=127.6.151.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:19:40.399809 25760 heartbeater.cc:347] Connected to a master server at 127.6.151.126:41659
I20240502 08:19:40.400183 25760 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:40.401185 25760 heartbeater.cc:511] Master 127.6.151.126:41659 requested a full tablet report, sending...
I20240502 08:19:40.403081 24643 ts_manager.cc:194] Re-registered known tserver with Master: c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255)
I20240502 08:19:40.404141 24643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.69:48731
W20240502 08:19:40.756930 25764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:19:40.757285 25764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:19:40.757443 25764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:19:40.765836 25764 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:19:40.765985 25764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:19:40.766111 25764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.151.68
I20240502 08:19:40.776134 25764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.151.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.151.68:43407
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.151.68
--webserver_port=36121
--tserver_master_addrs=127.6.151.126:41659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.151.68
--log_dir=/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true

Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 07:43:23 UTC on efd1337a2aa3
build id 87
ASAN enabled
I20240502 08:19:40.777550 25764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:19:40.779501 25764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:19:40.792874 25772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:40.793155 25769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:19:40.793588 25770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:19:40.794406 25764 server_base.cc:1034] running on GCE node
I20240502 08:19:40.795262 25764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:19:40.796267 25764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:19:40.797502 25764 hybrid_clock.cc:648] HybridClock initialized: now 1714637980797435 us; error 50 us; skew 500 ppm
I20240502 08:19:40.797852 25764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 08:19:40.800290 25764 webserver.cc:460] Webserver started at http://127.6.151.68:36121/ using document root <none> and password file <none>
I20240502 08:19:40.801051 25764 fs_manager.cc:359] Metadata directory not provided
I20240502 08:19:40.801196 25764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:19:40.806738 25764 fs_manager.cc:709] Time spent opening directory manager: real 0.003s	user 0.003s	sys 0.001s
I20240502 08:19:40.809579 25778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:40.810940 25764 fs_manager.cc:725] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20240502 08:19:40.811100 25764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "40fa9dc1519b476d813eb3f71e25d73c"
format_stamp: "Formatted at 2024-05-02 08:19:30 on dist-test-slave-x3qn"
I20240502 08:19:40.811909 25764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:19:40.838162 25764 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20240502 08:19:40.839092 25764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:19:40.839373 25764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:19:40.840348 25764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:19:40.841884 25764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:19:40.841971 25764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:40.842103 25764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:19:40.842159 25764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20240502 08:19:40.874986 25764 rpc_server.cc:307] RPC server started. Bound to: 127.6.151.68:43407
I20240502 08:19:40.875005 25890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.151.68:43407 every 8 connection(s)
I20240502 08:19:40.876860 25764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:19:40.878901  6749 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu as pid 25764
I20240502 08:19:40.892782 25891 heartbeater.cc:347] Connected to a master server at 127.6.151.126:41659
I20240502 08:19:40.893126 25891 heartbeater.cc:464] Registering TS with master...
I20240502 08:19:40.894006 25891 heartbeater.cc:511] Master 127.6.151.126:41659 requested a full tablet report, sending...
I20240502 08:19:40.895730 24643 ts_manager.cc:194] Re-registered known tserver with Master: 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407)
I20240502 08:19:40.896929 24643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.151.68:52587
I20240502 08:19:41.382827 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.400849 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:41.407042 25760 heartbeater.cc:503] Master 127.6.151.126:41659 was elected leader, sending a full tablet report...
I20240502 08:19:41.415416 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:41.417618 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:41.435808 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:41.586004 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.586867 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:41.591244 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:41.608016 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:41.609064 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:41.805094 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.806931 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:41.810956 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:41.815833 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:41.816242 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:41.899912 25891 heartbeater.cc:503] Master 127.6.151.126:41659 was elected leader, sending a full tablet report...
I20240502 08:19:41.998109 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:41.999064 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:41.999456 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:42.001134 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:42.011126 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:42.095147 25050 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.098021 24918 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.103559 24787 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.104456 25611 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.105247 25611 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:19:42.113461 24984 consensus_peers.cc:485] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd -> Peer 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407): Couldn't send request to peer 40fa9dc1519b476d813eb3f71e25d73c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11d0289cfeaa4c30a22c499c92b5b653. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:42.115327 26040 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.116755 24918 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.116149 24787 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.123529 24643 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:42.137061 25050 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.139442 24918 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.139886 24787 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.140432 26040 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.140717 26038 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.143906 26040 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.144280 24918 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
W20240502 08:19:42.147257 24983 consensus_peers.cc:485] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17d1d1592ecc4d52addd77be44075a1b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:42.148373 24787 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.149497 24641 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.164722 24918 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.167789 24787 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.168677 26063 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.169229 25050 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.170022 26063 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.172031 25543 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.173242 24787 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
W20240502 08:19:42.177618 24853 consensus_peers.cc:485] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 -> Peer 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407): Couldn't send request to peer 40fa9dc1519b476d813eb3f71e25d73c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 614c1e62e62e401baad4df6d5270ab57. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:42.177330 25050 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.179420 24640 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:42.196020 24787 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.199146 24918 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.200130 25050 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.200971 25600 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.201938 25600 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.205327 25600 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.206192 24918 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.206254 25050 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
W20240502 08:19:42.210951 24720 consensus_peers.cc:485] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4efbd31bd574cb1902d6856c4da2171. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:42.211849 24643 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.225916 24918 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.228468 25050 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:19:42.228683 24853 consensus_peers.cc:485] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 -> Peer 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407): Couldn't send request to peer 40fa9dc1519b476d813eb3f71e25d73c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf7e142110e04e9db0fdc2c5138117ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:42.228776 24787 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.229065 26063 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.229427 26043 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.232354 25543 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.234201 24787 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.236418 25050 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.239763 24640 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index -1 to 4, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:42.253544 24787 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.256024 25050 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:19:42.256191 24720 consensus_peers.cc:485] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d915caab5c694338a1a2a8ec599e1848. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:42.256249 24918 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:42.256695 26074 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.257234 26073 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:42.259763 26074 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.260637 25050 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.260756 24918 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:42.264667 24641 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index -1 to 4, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:42.533255 26129 ts_tablet_manager.cc:902] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:42.539422 26129 tablet_copy_client.cc:308] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:45829
I20240502 08:19:42.547938 25070 tablet_copy_service.cc:140] P b4fd6df034eb4418ab29ad24a7a8a9fd: Received BeginTabletCopySession request for tablet 17d1d1592ecc4d52addd77be44075a1b from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:56025)
I20240502 08:19:42.548195 25070 tablet_copy_service.cc:161] P b4fd6df034eb4418ab29ad24a7a8a9fd: Beginning new tablet copy session on tablet 17d1d1592ecc4d52addd77be44075a1b from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:56025: session id = c809bcf4be094d1b9a931b8431fdc60c-17d1d1592ecc4d52addd77be44075a1b
I20240502 08:19:42.550895 25070 tablet_copy_source_session.cc:205] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:42.555889 26129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d1d1592ecc4d52addd77be44075a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:42.569438 26129 tablet_copy_client.cc:783] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:42.569818 26129 tablet_copy_client.cc:647] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:42.573504 26129 tablet_copy_client.cc:523] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:42.577371 26156 ts_tablet_manager.cc:902] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: Initiating tablet copy from peer b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:42.577872 26129 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:42.596726 26129 log.cc:824] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:42.607646 26155 ts_tablet_manager.cc:902] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:42.633579 26156 tablet_copy_client.cc:308] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:45829
I20240502 08:19:42.635474 26155 tablet_copy_client.cc:308] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:34967
I20240502 08:19:42.644472 24807 tablet_copy_service.cc:140] P 3ccdae65238f49c7aa130138a4dec610: Received BeginTabletCopySession request for tablet a4efbd31bd574cb1902d6856c4da2171 from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:55325)
I20240502 08:19:42.644850 24807 tablet_copy_service.cc:161] P 3ccdae65238f49c7aa130138a4dec610: Beginning new tablet copy session on tablet a4efbd31bd574cb1902d6856c4da2171 from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:55325: session id = c809bcf4be094d1b9a931b8431fdc60c-a4efbd31bd574cb1902d6856c4da2171
I20240502 08:19:42.649485 24807 tablet_copy_source_session.cc:205] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:42.652707 26155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efbd31bd574cb1902d6856c4da2171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:42.656040 25070 tablet_copy_service.cc:140] P b4fd6df034eb4418ab29ad24a7a8a9fd: Received BeginTabletCopySession request for tablet 11d0289cfeaa4c30a22c499c92b5b653 from peer 40fa9dc1519b476d813eb3f71e25d73c ({username='slave'} at 127.6.151.68:57367)
I20240502 08:19:42.656257 25070 tablet_copy_service.cc:161] P b4fd6df034eb4418ab29ad24a7a8a9fd: Beginning new tablet copy session on tablet 11d0289cfeaa4c30a22c499c92b5b653 from peer 40fa9dc1519b476d813eb3f71e25d73c at {username='slave'} at 127.6.151.68:57367: session id = 40fa9dc1519b476d813eb3f71e25d73c-11d0289cfeaa4c30a22c499c92b5b653
I20240502 08:19:42.659286 25070 tablet_copy_source_session.cc:205] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:42.660168 26155 tablet_copy_client.cc:783] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:42.660518 26155 tablet_copy_client.cc:647] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:42.662936 26155 tablet_copy_client.cc:523] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:42.665202 26156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d0289cfeaa4c30a22c499c92b5b653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:42.671514 26155 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:42.672833 26129 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:42.673498 26129 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:42.674078 26129 ts_tablet_manager.cc:1367] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.097s	user 0.076s	sys 0.009s
I20240502 08:19:42.680972 26129 raft_consensus.cc:348] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.681653 26129 raft_consensus.cc:729] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:42.682264 26129 consensus_queue.cc:260] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.688115 26129 ts_tablet_manager.cc:1398] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.004s
I20240502 08:19:42.689687 26175 ts_tablet_manager.cc:902] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: Initiating tablet copy from peer e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:42.689741 25070 tablet_copy_service.cc:342] P b4fd6df034eb4418ab29ad24a7a8a9fd: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-17d1d1592ecc4d52addd77be44075a1b received from {username='slave'} at 127.6.151.69:56025
I20240502 08:19:42.689916 25070 tablet_copy_service.cc:433] P b4fd6df034eb4418ab29ad24a7a8a9fd: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-17d1d1592ecc4d52addd77be44075a1b on tablet 17d1d1592ecc4d52addd77be44075a1b with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:42.691895 26156 tablet_copy_client.cc:783] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:42.692623 26156 tablet_copy_client.cc:647] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:42.695405 26156 tablet_copy_client.cc:523] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:42.702919 26175 tablet_copy_client.cc:308] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:35329
I20240502 08:19:42.705015 26173 ts_tablet_manager.cc:902] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: Initiating tablet copy from peer e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:42.707177 26156 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap starting.
I20240502 08:19:42.711161 24939 tablet_copy_service.cc:140] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Received BeginTabletCopySession request for tablet 614c1e62e62e401baad4df6d5270ab57 from peer 40fa9dc1519b476d813eb3f71e25d73c ({username='slave'} at 127.6.151.68:37637)
I20240502 08:19:42.711411 24939 tablet_copy_service.cc:161] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Beginning new tablet copy session on tablet 614c1e62e62e401baad4df6d5270ab57 from peer 40fa9dc1519b476d813eb3f71e25d73c at {username='slave'} at 127.6.151.68:37637: session id = 40fa9dc1519b476d813eb3f71e25d73c-614c1e62e62e401baad4df6d5270ab57
I20240502 08:19:42.715148 24939 tablet_copy_source_session.cc:205] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:42.718415 26175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c1e62e62e401baad4df6d5270ab57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:42.725039 26156 log.cc:824] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: Log is configured to *not* fsync() on all Append() calls
I20240502 08:19:42.725912 26175 tablet_copy_client.cc:783] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:42.726276 26175 tablet_copy_client.cc:647] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:42.729267 26175 tablet_copy_client.cc:523] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:42.733085 26173 tablet_copy_client.cc:308] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:35329
I20240502 08:19:42.733362 26175 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap starting.
I20240502 08:19:42.733995 24939 tablet_copy_service.cc:140] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Received BeginTabletCopySession request for tablet cf7e142110e04e9db0fdc2c5138117ae from peer 40fa9dc1519b476d813eb3f71e25d73c ({username='slave'} at 127.6.151.68:37637)
I20240502 08:19:42.734208 24939 tablet_copy_service.cc:161] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Beginning new tablet copy session on tablet cf7e142110e04e9db0fdc2c5138117ae from peer 40fa9dc1519b476d813eb3f71e25d73c at {username='slave'} at 127.6.151.68:37637: session id = 40fa9dc1519b476d813eb3f71e25d73c-cf7e142110e04e9db0fdc2c5138117ae
I20240502 08:19:42.737098 24939 tablet_copy_source_session.cc:205] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:42.738706 26173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7e142110e04e9db0fdc2c5138117ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:42.743901 26155 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:42.744520 26155 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:42.744935 26155 ts_tablet_manager.cc:1367] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.074s	user 0.045s	sys 0.010s
I20240502 08:19:42.746605 26173 tablet_copy_client.cc:783] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:42.746423 26155 raft_consensus.cc:348] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.746778 26155 raft_consensus.cc:729] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:42.746991 26173 tablet_copy_client.cc:647] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:42.747107 26155 consensus_queue.cc:260] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.747881 26155 ts_tablet_manager.cc:1398] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:42.748981 26173 tablet_copy_client.cc:523] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:42.749352 24807 tablet_copy_service.cc:342] P 3ccdae65238f49c7aa130138a4dec610: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-a4efbd31bd574cb1902d6856c4da2171 received from {username='slave'} at 127.6.151.69:55325
I20240502 08:19:42.749543 24807 tablet_copy_service.cc:433] P 3ccdae65238f49c7aa130138a4dec610: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-a4efbd31bd574cb1902d6856c4da2171 on tablet a4efbd31bd574cb1902d6856c4da2171 with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:42.753350 26173 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap starting.
I20240502 08:19:42.780845 26175 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:42.781428 26175 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap complete.
I20240502 08:19:42.781916 26175 ts_tablet_manager.cc:1367] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent bootstrapping tablet: real 0.049s	user 0.032s	sys 0.012s
I20240502 08:19:42.786386 26156 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:42.786968 26156 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap complete.
I20240502 08:19:42.787262 26156 ts_tablet_manager.cc:1367] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent bootstrapping tablet: real 0.080s	user 0.066s	sys 0.007s
I20240502 08:19:42.787415 26175 raft_consensus.cc:348] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.787892 26175 raft_consensus.cc:729] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Initialized, Role: LEARNER
I20240502 08:19:42.788295 26156 raft_consensus.cc:348] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.788388 26175 consensus_queue.cc:260] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.788574 26156 raft_consensus.cc:729] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Initialized, Role: LEARNER
I20240502 08:19:42.788830 26156 consensus_queue.cc:260] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.790266 26175 ts_tablet_manager.cc:1398] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20240502 08:19:42.791481 24939 tablet_copy_service.cc:342] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Request end of tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-614c1e62e62e401baad4df6d5270ab57 received from {username='slave'} at 127.6.151.68:37637
I20240502 08:19:42.791641 24939 tablet_copy_service.cc:433] P e2cb3e9aae7d4ef7b06b5d1e6501add5: ending tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-614c1e62e62e401baad4df6d5270ab57 on tablet 614c1e62e62e401baad4df6d5270ab57 with peer 40fa9dc1519b476d813eb3f71e25d73c
I20240502 08:19:42.795276 26155 ts_tablet_manager.cc:902] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:42.796258 26156 ts_tablet_manager.cc:1398] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.007s
I20240502 08:19:42.797577 25070 tablet_copy_service.cc:342] P b4fd6df034eb4418ab29ad24a7a8a9fd: Request end of tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-11d0289cfeaa4c30a22c499c92b5b653 received from {username='slave'} at 127.6.151.68:57367
I20240502 08:19:42.797727 25070 tablet_copy_service.cc:433] P b4fd6df034eb4418ab29ad24a7a8a9fd: ending tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-11d0289cfeaa4c30a22c499c92b5b653 on tablet 11d0289cfeaa4c30a22c499c92b5b653 with peer 40fa9dc1519b476d813eb3f71e25d73c
I20240502 08:19:42.803330 26155 tablet_copy_client.cc:308] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:34967
I20240502 08:19:42.804395 24807 tablet_copy_service.cc:140] P 3ccdae65238f49c7aa130138a4dec610: Received BeginTabletCopySession request for tablet d915caab5c694338a1a2a8ec599e1848 from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:55325)
I20240502 08:19:42.804560 24807 tablet_copy_service.cc:161] P 3ccdae65238f49c7aa130138a4dec610: Beginning new tablet copy session on tablet d915caab5c694338a1a2a8ec599e1848 from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:55325: session id = c809bcf4be094d1b9a931b8431fdc60c-d915caab5c694338a1a2a8ec599e1848
I20240502 08:19:42.806720 24807 tablet_copy_source_session.cc:205] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:42.807624 26173 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:42.808022 26173 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap complete.
I20240502 08:19:42.808092 26155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d915caab5c694338a1a2a8ec599e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:42.808291 26173 ts_tablet_manager.cc:1367] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: Time spent bootstrapping tablet: real 0.055s	user 0.046s	sys 0.005s
I20240502 08:19:42.808936 26173 raft_consensus.cc:348] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.809154 26173 raft_consensus.cc:729] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Initialized, Role: LEARNER
I20240502 08:19:42.809340 26173 consensus_queue.cc:260] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:42.809808 26173 ts_tablet_manager.cc:1398] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:42.810701 24939 tablet_copy_service.cc:342] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Request end of tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-cf7e142110e04e9db0fdc2c5138117ae received from {username='slave'} at 127.6.151.68:37637
I20240502 08:19:42.810838 24939 tablet_copy_service.cc:433] P e2cb3e9aae7d4ef7b06b5d1e6501add5: ending tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-cf7e142110e04e9db0fdc2c5138117ae on tablet cf7e142110e04e9db0fdc2c5138117ae with peer 40fa9dc1519b476d813eb3f71e25d73c
I20240502 08:19:42.813863 26155 tablet_copy_client.cc:783] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:42.814189 26155 tablet_copy_client.cc:647] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:42.815786 26155 tablet_copy_client.cc:523] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:42.818257 26155 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:42.848685 26155 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:42.849136 26155 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:42.849397 26155 ts_tablet_manager.cc:1367] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.031s	user 0.031s	sys 0.000s
I20240502 08:19:42.850060 26155 raft_consensus.cc:348] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.850240 26155 raft_consensus.cc:729] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:42.850457 26155 consensus_queue.cc:260] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:42.851042 26155 ts_tablet_manager.cc:1398] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.004s
I20240502 08:19:42.852624 24807 tablet_copy_service.cc:342] P 3ccdae65238f49c7aa130138a4dec610: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-d915caab5c694338a1a2a8ec599e1848 received from {username='slave'} at 127.6.151.69:55325
I20240502 08:19:42.852748 24807 tablet_copy_service.cc:433] P 3ccdae65238f49c7aa130138a4dec610: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-d915caab5c694338a1a2a8ec599e1848 on tablet d915caab5c694338a1a2a8ec599e1848 with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:43.001686 25715 raft_consensus.cc:1203] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:43.006291 25846 raft_consensus.cc:1203] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:43.065652 25715 raft_consensus.cc:1203] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:43.182895 25846 raft_consensus.cc:1203] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:43.184577 25715 raft_consensus.cc:1203] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:43.303471 25846 raft_consensus.cc:1203] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4]   Dedup: 1.4->[]
I20240502 08:19:43.501564 26226 raft_consensus.cc:1050] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:43.502403 26226 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:43.504953 24918 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.505095 24786 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.505255 25715 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.505693 26226 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.506089 26292 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.506390 26225 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.512403 26225 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.513361 25715 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.513846 24786 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.517160 24918 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.518410 24640 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.525362 25050 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:43.527217 25715 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.527390 24786 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.527951 26225 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.528213 26226 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.530396 26040 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.531071 24786 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.531267 25715 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.533965 24631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 17d1d1592ecc4d52addd77be44075a1b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:43.535854 24639 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.540184 24898 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d1d1592ecc4d52addd77be44075a1b with delete_type TABLET_DATA_TOMBSTONED (TS e2cb3e9aae7d4ef7b06b5d1e6501add5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53540
I20240502 08:19:43.541129 26300 tablet_replica.cc:330] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: stopping tablet replica
I20240502 08:19:43.541664 26300 raft_consensus.cc:2229] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.542167 26300 raft_consensus.cc:2258] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.543612 26300 ts_tablet_manager.cc:1875] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.548444 26300 ts_tablet_manager.cc:1888] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:43.548578 26300 log.cc:1196] T 17d1d1592ecc4d52addd77be44075a1b P e2cb3e9aae7d4ef7b06b5d1e6501add5: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/17d1d1592ecc4d52addd77be44075a1b
I20240502 08:19:43.549479 24631 catalog_manager.cc:4830] TS e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329): tablet 17d1d1592ecc4d52addd77be44075a1b (table rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]) successfully deleted
I20240502 08:19:43.584028 26040 raft_consensus.cc:1050] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: attempting to promote NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c to VOTER
I20240502 08:19:43.584909 26040 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:43.587697 24786 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.587728 24918 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.587754 25846 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.588310 26292 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.588708 26040 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.589020 26225 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.597875 26228 raft_consensus.cc:1050] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:43.599522 26228 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:43.606325 25050 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.607128 26227 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.607313 25715 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.607503 26225 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.608137 26227 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.610909 24917 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.615415 24640 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.615551 25846 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.619925 24786 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.620898 24917 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.621583 26227 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.623075 25050 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:43.630558 26227 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.633555 24918 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.634130 26225 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.635337 25845 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.635125 25714 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.637653 26292 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.638485 26226 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.640971 25845 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.641057 24918 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.642360 24631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11d0289cfeaa4c30a22c499c92b5b653 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:43.642887 24639 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.648599 25050 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.659125 26228 raft_consensus.cc:1050] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:43.659986 26228 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:43.644026 24640 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.664144 24917 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.665542 25715 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.665717 24918 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.666196 26310 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.666474 25050 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.667282 26074 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.669359 26074 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.669605 24767 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d0289cfeaa4c30a22c499c92b5b653 with delete_type TABLET_DATA_TOMBSTONED (TS 3ccdae65238f49c7aa130138a4dec610 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59132
I20240502 08:19:43.671291 26316 tablet_replica.cc:330] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: stopping tablet replica
I20240502 08:19:43.671797 26316 raft_consensus.cc:2229] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.672498 26316 raft_consensus.cc:2258] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.674470 26316 ts_tablet_manager.cc:1875] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.677733 24917 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.676781 26309 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.679545 24786 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:43.682209 25714 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.683120 26310 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.683262 25050 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.684320 26310 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.686014 26310 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.686412 25049 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.686879 25714 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.690114 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d915caab5c694338a1a2a8ec599e1848 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:43.692822 25050 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.694314 25715 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.700589 26316 ts_tablet_manager.cc:1888] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:43.700891 26316 log.cc:1196] T 11d0289cfeaa4c30a22c499c92b5b653 P 3ccdae65238f49c7aa130138a4dec610: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/11d0289cfeaa4c30a22c499c92b5b653
I20240502 08:19:43.702095 24630 catalog_manager.cc:4830] TS 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967): tablet 11d0289cfeaa4c30a22c499c92b5b653 (table rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]) successfully deleted
I20240502 08:19:43.703857 24643 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.704660 24643 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 4 to 5, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.710196 24898 tablet_service.cc:1514] Processing DeleteTablet for tablet d915caab5c694338a1a2a8ec599e1848 with delete_type TABLET_DATA_TOMBSTONED (TS e2cb3e9aae7d4ef7b06b5d1e6501add5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53540
I20240502 08:19:43.710886 26300 tablet_replica.cc:330] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: stopping tablet replica
I20240502 08:19:43.711112 26300 raft_consensus.cc:2229] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.711212 24786 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:43.711421 26300 raft_consensus.cc:2258] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.712610 26300 ts_tablet_manager.cc:1875] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.713317 25050 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.713979 26309 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.717089 25715 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.716895 26310 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.717787 26228 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.717723 25050 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.719440 25715 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:43.721915 24640 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 5 to 6, VOTER e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66) evicted. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:43.722204 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4efbd31bd574cb1902d6856c4da2171 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:43.725056 26255 raft_consensus.cc:1050] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: attempting to promote NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c to VOTER
I20240502 08:19:43.725857 26255 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:43.730803 24898 tablet_service.cc:1514] Processing DeleteTablet for tablet a4efbd31bd574cb1902d6856c4da2171 with delete_type TABLET_DATA_TOMBSTONED (TS e2cb3e9aae7d4ef7b06b5d1e6501add5 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53540
I20240502 08:19:43.734354 26300 ts_tablet_manager.cc:1888] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:43.734510 26300 log.cc:1196] T d915caab5c694338a1a2a8ec599e1848 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/d915caab5c694338a1a2a8ec599e1848
I20240502 08:19:43.735205 26300 tablet_replica.cc:330] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: stopping tablet replica
I20240502 08:19:43.735257 24631 catalog_manager.cc:4830] TS e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329): tablet d915caab5c694338a1a2a8ec599e1848 (table rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]) successfully deleted
I20240502 08:19:43.735414 26300 raft_consensus.cc:2229] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.735663 26300 raft_consensus.cc:2258] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.736490 26300 ts_tablet_manager.cc:1875] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.742058 26300 ts_tablet_manager.cc:1888] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:43.742108 25050 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.742230 26300 log.cc:1196] T a4efbd31bd574cb1902d6856c4da2171 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4efbd31bd574cb1902d6856c4da2171
I20240502 08:19:43.742771 26043 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.743096 25845 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.743583 24631 catalog_manager.cc:4830] TS e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329): tablet a4efbd31bd574cb1902d6856c4da2171 (table rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]) successfully deleted
I20240502 08:19:43.744148 24786 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.744254 26043 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.744899 26043 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.749821 25050 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.749073 26255 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.750016 25845 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.755183 24786 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.757421 24640 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.766085 24917 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:43.768568 25845 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.769388 26255 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.770300 25050 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.771190 26255 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.772459 26254 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.774403 25050 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.777115 25845 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.777755 24631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf7e142110e04e9db0fdc2c5138117ae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:43.780961 24642 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.786103 24767 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7e142110e04e9db0fdc2c5138117ae with delete_type TABLET_DATA_TOMBSTONED (TS 3ccdae65238f49c7aa130138a4dec610 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59132
I20240502 08:19:43.786569 26316 tablet_replica.cc:330] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: stopping tablet replica
I20240502 08:19:43.786811 26316 raft_consensus.cc:2229] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.787184 26316 raft_consensus.cc:2258] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.788287 26316 ts_tablet_manager.cc:1875] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.793661 26316 ts_tablet_manager.cc:1888] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:43.793854 26316 log.cc:1196] T cf7e142110e04e9db0fdc2c5138117ae P 3ccdae65238f49c7aa130138a4dec610: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf7e142110e04e9db0fdc2c5138117ae
I20240502 08:19:43.794611 24630 catalog_manager.cc:4830] TS 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967): tablet cf7e142110e04e9db0fdc2c5138117ae (table rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]) successfully deleted
I20240502 08:19:43.865239 26254 raft_consensus.cc:1050] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: attempting to promote NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c to VOTER
I20240502 08:19:43.865988 26254 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:43.868439 25050 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.868446 24786 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.868468 25845 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:43.869103 26255 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.869438 26254 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.869740 26328 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:43.873361 26328 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.874086 25845 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.879107 24643 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.880496 25050 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.883196 24786 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } attrs { replace: true } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.886880 24917 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:43.889209 25845 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.890486 26255 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.890866 25050 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:43.891393 26255 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:43.892973 26255 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.895902 24631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 614c1e62e62e401baad4df6d5270ab57 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:43.896737 25845 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.897348 25050 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:43.898993 24639 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 5 to 6, VOTER 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65) evicted. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:43.903141 24767 tablet_service.cc:1514] Processing DeleteTablet for tablet 614c1e62e62e401baad4df6d5270ab57 with delete_type TABLET_DATA_TOMBSTONED (TS 3ccdae65238f49c7aa130138a4dec610 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59132
I20240502 08:19:43.903546 26316 tablet_replica.cc:330] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: stopping tablet replica
I20240502 08:19:43.903750 26316 raft_consensus.cc:2229] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:43.904079 26316 raft_consensus.cc:2258] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:43.904881 26316 ts_tablet_manager.cc:1875] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:43.909863 26316 ts_tablet_manager.cc:1888] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:19:43.909973 26316 log.cc:1196] T 614c1e62e62e401baad4df6d5270ab57 P 3ccdae65238f49c7aa130138a4dec610: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-0/wal/wals/614c1e62e62e401baad4df6d5270ab57
I20240502 08:19:43.910673 24630 catalog_manager.cc:4830] TS 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967): tablet 614c1e62e62e401baad4df6d5270ab57 (table rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]) successfully deleted
I20240502 08:19:44.154438 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:44.163520 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:44.164695 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:44.181847 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:44.184360 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:44.366242 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:44.368395 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:44.368801 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:44.372741 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:44.373718 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:44.545881 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:44.550604 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:44.553449 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:44.562438 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:44.568737 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:45.187467 24642 ts_manager.cc:292] Set tserver state for b4fd6df034eb4418ab29ad24a7a8a9fd to MAINTENANCE_MODE
I20240502 08:19:45.771564 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:45.797722 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:45.808842 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:45.810608 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:45.811496 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:45.989562 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:45.996963 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:45.999389 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:46.015375 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:46.033144 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:46.220932 25826 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:46.225881 25030 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:19:46.226255 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:46.226275 25695 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:46.239040 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:46.309556 24786 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:46.311928 25715 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:46.312219 25050 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:46.312546 26602 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:46.312875 26309 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:46.315768 26309 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:46.316781 25715 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:46.319530 25050 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:46.321463 24643 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:46.329361 24786 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:46.333567 25715 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:46.334350 26602 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:46.336258 25050 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:46.337909 26605 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:46.338310 26606 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:46.339660 25050 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:46.342959 24630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a4efbd31bd574cb1902d6856c4da2171 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:46.343966 25715 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:46.345185 24643 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:46.351228 24917 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
W20240502 08:19:46.352701 24721 consensus_peers.cc:485] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 -> Peer 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407): Couldn't send request to peer 40fa9dc1519b476d813eb3f71e25d73c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4efbd31bd574cb1902d6856c4da2171. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:46.353911 25050 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:46.353911 25845 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:46.354539 26255 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:46.355365 26255 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:46.358080 26255 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:46.358844 25845 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:46.366750 25050 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:46.367375 24642 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:46.375643 24917 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:46.380587 25050 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:46.381579 25845 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:46.381507 26255 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:46.382355 26622 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:46.385238 26255 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:46.386404 25050 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
W20240502 08:19:46.389107 24852 consensus_peers.cc:485] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 614c1e62e62e401baad4df6d5270ab57. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:46.389834 24631 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 614c1e62e62e401baad4df6d5270ab57 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:46.392681 24639 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:46.410470 25845 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:46.421649 25050 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:46.426358 25715 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:46.427027 24786 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:46.427124 26633 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:46.427758 26490 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:46.431139 26490 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:46.431828 24786 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:46.435755 25715 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:46.437677 24643 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:46.445706 25050 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:46.449398 24786 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:46.450086 25715 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:46.450140 26633 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:19:46.451284 24984 consensus_peers.cc:485] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd -> Peer 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407): Couldn't send request to peer 40fa9dc1519b476d813eb3f71e25d73c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 17d1d1592ecc4d52addd77be44075a1b. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:46.451588 26490 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:46.453318 26633 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:46.454160 24786 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:46.455359 25715 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:46.460666 24643 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:46.461256 24631 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 17d1d1592ecc4d52addd77be44075a1b with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:46.837555 26668 ts_tablet_manager.cc:902] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: Initiating tablet copy from peer b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:46.839133 26668 tablet_copy_client.cc:308] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:45829
I20240502 08:19:46.839960 25070 tablet_copy_service.cc:140] P b4fd6df034eb4418ab29ad24a7a8a9fd: Received BeginTabletCopySession request for tablet 17d1d1592ecc4d52addd77be44075a1b from peer 40fa9dc1519b476d813eb3f71e25d73c ({username='slave'} at 127.6.151.68:57367)
I20240502 08:19:46.840157 25070 tablet_copy_service.cc:161] P b4fd6df034eb4418ab29ad24a7a8a9fd: Beginning new tablet copy session on tablet 17d1d1592ecc4d52addd77be44075a1b from peer 40fa9dc1519b476d813eb3f71e25d73c at {username='slave'} at 127.6.151.68:57367: session id = 40fa9dc1519b476d813eb3f71e25d73c-17d1d1592ecc4d52addd77be44075a1b
I20240502 08:19:46.842458 25070 tablet_copy_source_session.cc:205] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:46.843832 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17d1d1592ecc4d52addd77be44075a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:46.848462 26668 tablet_copy_client.cc:783] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:46.848716 26668 tablet_copy_client.cc:647] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:46.850663 26668 tablet_copy_client.cc:523] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:46.853770 26668 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap starting.
I20240502 08:19:46.888984 26668 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:46.889534 26668 tablet_bootstrap.cc:492] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap complete.
I20240502 08:19:46.889907 26668 ts_tablet_manager.cc:1367] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: Time spent bootstrapping tablet: real 0.036s	user 0.033s	sys 0.005s
I20240502 08:19:46.890630 26668 raft_consensus.cc:348] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:46.890893 26668 raft_consensus.cc:729] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Initialized, Role: LEARNER
I20240502 08:19:46.891079 26668 consensus_queue.cc:260] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:46.891609 26668 ts_tablet_manager.cc:1398] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c: Time spent starting tablet: real 0.002s	user 0.001s	sys 0.000s
I20240502 08:19:46.892624 25070 tablet_copy_service.cc:342] P b4fd6df034eb4418ab29ad24a7a8a9fd: Request end of tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-17d1d1592ecc4d52addd77be44075a1b received from {username='slave'} at 127.6.151.68:57367
I20240502 08:19:46.892818 25070 tablet_copy_service.cc:433] P b4fd6df034eb4418ab29ad24a7a8a9fd: ending tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-17d1d1592ecc4d52addd77be44075a1b on tablet 17d1d1592ecc4d52addd77be44075a1b with peer 40fa9dc1519b476d813eb3f71e25d73c
I20240502 08:19:46.894790 26668 ts_tablet_manager.cc:902] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: Initiating tablet copy from peer 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:46.896207 26668 tablet_copy_client.cc:308] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:34967
I20240502 08:19:46.901846 24807 tablet_copy_service.cc:140] P 3ccdae65238f49c7aa130138a4dec610: Received BeginTabletCopySession request for tablet a4efbd31bd574cb1902d6856c4da2171 from peer 40fa9dc1519b476d813eb3f71e25d73c ({username='slave'} at 127.6.151.68:39129)
I20240502 08:19:46.902019 24807 tablet_copy_service.cc:161] P 3ccdae65238f49c7aa130138a4dec610: Beginning new tablet copy session on tablet a4efbd31bd574cb1902d6856c4da2171 from peer 40fa9dc1519b476d813eb3f71e25d73c at {username='slave'} at 127.6.151.68:39129: session id = 40fa9dc1519b476d813eb3f71e25d73c-a4efbd31bd574cb1902d6856c4da2171
I20240502 08:19:46.904032 24807 tablet_copy_source_session.cc:205] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:46.905349 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efbd31bd574cb1902d6856c4da2171. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:46.909675 26668 tablet_copy_client.cc:783] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:46.909924 26668 tablet_copy_client.cc:647] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:46.911717 26668 tablet_copy_client.cc:523] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:46.914057 26668 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap starting.
I20240502 08:19:46.942729 26692 ts_tablet_manager.cc:902] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:46.963044 26692 tablet_copy_client.cc:308] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:35329
I20240502 08:19:46.969174 26668 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:46.969653 26668 tablet_bootstrap.cc:492] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap complete.
I20240502 08:19:46.969954 26668 ts_tablet_manager.cc:1367] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent bootstrapping tablet: real 0.056s	user 0.049s	sys 0.004s
I20240502 08:19:46.969975 24939 tablet_copy_service.cc:140] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Received BeginTabletCopySession request for tablet 614c1e62e62e401baad4df6d5270ab57 from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:51327)
I20240502 08:19:46.970160 24939 tablet_copy_service.cc:161] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Beginning new tablet copy session on tablet 614c1e62e62e401baad4df6d5270ab57 from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:51327: session id = c809bcf4be094d1b9a931b8431fdc60c-614c1e62e62e401baad4df6d5270ab57
I20240502 08:19:46.970710 26668 raft_consensus.cc:348] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:46.970918 26668 raft_consensus.cc:729] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Initialized, Role: LEARNER
I20240502 08:19:46.971112 26668 consensus_queue.cc:260] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:46.971570 26668 ts_tablet_manager.cc:1398] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 08:19:46.972285 24807 tablet_copy_service.cc:342] P 3ccdae65238f49c7aa130138a4dec610: Request end of tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-a4efbd31bd574cb1902d6856c4da2171 received from {username='slave'} at 127.6.151.68:39129
I20240502 08:19:46.972432 24807 tablet_copy_service.cc:433] P 3ccdae65238f49c7aa130138a4dec610: ending tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-a4efbd31bd574cb1902d6856c4da2171 on tablet a4efbd31bd574cb1902d6856c4da2171 with peer 40fa9dc1519b476d813eb3f71e25d73c
I20240502 08:19:46.972586 24939 tablet_copy_source_session.cc:205] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:46.973909 26692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 614c1e62e62e401baad4df6d5270ab57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:46.978561 26692 tablet_copy_client.cc:783] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:46.978845 26692 tablet_copy_client.cc:647] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:46.980732 26692 tablet_copy_client.cc:523] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:46.983457 26692 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:47.015452 26692 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:47.015949 26692 tablet_bootstrap.cc:492] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:47.016250 26692 ts_tablet_manager.cc:1367] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.033s	user 0.029s	sys 0.002s
I20240502 08:19:47.016995 26692 raft_consensus.cc:348] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:47.017225 26692 raft_consensus.cc:729] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:47.017423 26692 consensus_queue.cc:260] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:47.018167 26692 ts_tablet_manager.cc:1398] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.002s	user 0.002s	sys 0.001s
I20240502 08:19:47.019019 24939 tablet_copy_service.cc:342] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-614c1e62e62e401baad4df6d5270ab57 received from {username='slave'} at 127.6.151.69:51327
I20240502 08:19:47.019160 24939 tablet_copy_service.cc:433] P e2cb3e9aae7d4ef7b06b5d1e6501add5: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-614c1e62e62e401baad4df6d5270ab57 on tablet 614c1e62e62e401baad4df6d5270ab57 with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:47.372099 25845 raft_consensus.cc:1203] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:19:47.405927 25715 raft_consensus.cc:1203] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:19:47.482892 25845 raft_consensus.cc:1203] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:19:47.759677 26721 raft_consensus.cc:1050] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: attempting to promote NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c to VOTER
I20240502 08:19:47.760434 26721 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:47.762931 24786 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.762965 25715 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.762939 25845 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.763710 26757 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.764092 26721 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.764387 26639 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.767834 26757 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.768714 25715 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.773108 24786 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.773763 24639 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:47.779793 25845 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.875484 25050 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b"
dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45818
I20240502 08:19:47.875772 25050 raft_consensus.cc:595] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:47.889361 26739 raft_consensus.cc:1050] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610: attempting to promote NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c to VOTER
I20240502 08:19:47.890714 26739 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:47.894086 25715 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.894287 25845 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.894845 25050 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.894985 26606 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.896004 26606 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.896728 26606 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.911698 26740 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.918504 25845 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.918494 24643 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:47.921171 25050 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.928583 25715 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.931509 24786 consensus_queue.cc:237] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:47.933859 25845 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:47.933941 25715 raft_consensus.cc:1261] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:47.934621 26739 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:47.935050 26785 consensus_queue.cc:1035] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:47.937567 26739 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.940037 25845 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.941553 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a4efbd31bd574cb1902d6856c4da2171 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:47.942721 25715 raft_consensus.cc:2932] T a4efbd31bd574cb1902d6856c4da2171 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.945995 24642 catalog_manager.cc:5484] T a4efbd31bd574cb1902d6856c4da2171 P 40fa9dc1519b476d813eb3f71e25d73c reported cstate change: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:47.950513 26790 raft_consensus.cc:1050] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:47.950943 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet a4efbd31bd574cb1902d6856c4da2171 with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:47.951771 26790 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:47.951992 26791 tablet_replica.cc:330] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:47.952543 26791 raft_consensus.cc:2229] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:47.954521 25845 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.954775 25715 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.953146 26791 raft_consensus.cc:2258] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:47.954804 25050 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:47.955420 26729 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.956195 26729 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.956606 26793 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:47.956984 26791 ts_tablet_manager.cc:1875] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:47.968483 26791 ts_tablet_manager.cc:1888] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:19:47.968919 26791 log.cc:1196] T a4efbd31bd574cb1902d6856c4da2171 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/a4efbd31bd574cb1902d6856c4da2171
I20240502 08:19:47.970296 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet a4efbd31bd574cb1902d6856c4da2171 (table rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]) successfully deleted
I20240502 08:19:47.971544 26729 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.972419 25715 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.973039 25845 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.977252 24643 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.979162 25050 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.982331 24917 consensus_queue.cc:237] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:47.984608 25715 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:47.985556 25845 raft_consensus.cc:1261] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:47.985931 26797 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:47.986282 26790 consensus_queue.cc:1035] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:47.988807 26793 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.989478 25715 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.992383 24631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 614c1e62e62e401baad4df6d5270ab57 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:47.992539 25845 raft_consensus.cc:2932] T 614c1e62e62e401baad4df6d5270ab57 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.993570 24639 catalog_manager.cc:5484] T 614c1e62e62e401baad4df6d5270ab57 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:47.997638 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet 614c1e62e62e401baad4df6d5270ab57 with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:47.998016 26791 tablet_replica.cc:330] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:47.998232 26791 raft_consensus.cc:2229] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:47.998565 26791 raft_consensus.cc:2258] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:47.999468 26791 ts_tablet_manager.cc:1875] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:48.004174 26791 ts_tablet_manager.cc:1888] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:19:48.004287 26791 log.cc:1196] T 614c1e62e62e401baad4df6d5270ab57 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/614c1e62e62e401baad4df6d5270ab57
I20240502 08:19:48.005081 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet 614c1e62e62e401baad4df6d5270ab57 (table rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]) successfully deleted
I20240502 08:19:48.147768 25050 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b"
dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:45848
I20240502 08:19:48.147987 25050 raft_consensus.cc:595] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:48.249770 26721 raft_consensus.cc:979] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: : Instructing follower c809bcf4be094d1b9a931b8431fdc60c to start an election
I20240502 08:19:48.250015 26721 raft_consensus.cc:1067] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Signalling peer c809bcf4be094d1b9a931b8431fdc60c to start an election
I20240502 08:19:48.250936 25715 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b"
dest_uuid: "c809bcf4be094d1b9a931b8431fdc60c"
 from {username='slave'} at 127.6.151.67:58319
I20240502 08:19:48.251224 25715 raft_consensus.cc:482] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:48.251401 25715 raft_consensus.cc:3037] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:48.253881 25715 raft_consensus.cc:504] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:48.255508 25715 leader_election.cc:288] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [CANDIDATE]: Term 2 election: Requested vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829), 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407)
I20240502 08:19:48.256035 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b" candidate_uuid: "c809bcf4be094d1b9a931b8431fdc60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:48.256023 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b" candidate_uuid: "c809bcf4be094d1b9a931b8431fdc60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "3ccdae65238f49c7aa130138a4dec610"
I20240502 08:19:48.256242 24786 raft_consensus.cc:3037] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:48.256268 25050 raft_consensus.cc:3032] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:48.256390 25050 raft_consensus.cc:729] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Running, Role: LEADER
I20240502 08:19:48.256711 25050 consensus_queue.cc:260] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:48.257198 25050 raft_consensus.cc:3037] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:48.259290 24786 raft_consensus.cc:2454] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c809bcf4be094d1b9a931b8431fdc60c in term 2.
I20240502 08:19:48.259953 25050 raft_consensus.cc:2454] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c809bcf4be094d1b9a931b8431fdc60c in term 2.
I20240502 08:19:48.261583 25649 leader_election.cc:302] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, b4fd6df034eb4418ab29ad24a7a8a9fd, c809bcf4be094d1b9a931b8431fdc60c; no voters: 
I20240502 08:19:48.262066 26704 raft_consensus.cc:2781] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:48.262684 26704 raft_consensus.cc:686] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 2 LEADER]: Becoming Leader. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Running, Role: LEADER
I20240502 08:19:48.263217 26704 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:48.267838 25845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17d1d1592ecc4d52addd77be44075a1b" candidate_uuid: "c809bcf4be094d1b9a931b8431fdc60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "40fa9dc1519b476d813eb3f71e25d73c"
I20240502 08:19:48.267829 24643 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: term changed from 1 to 2, leader changed from b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) to c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69). New cstate: current_term: 2 leader_uuid: "c809bcf4be094d1b9a931b8431fdc60c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:48.268206 25845 raft_consensus.cc:3037] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:48.270912 25845 raft_consensus.cc:2454] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c809bcf4be094d1b9a931b8431fdc60c in term 2.
I20240502 08:19:48.380244 24786 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:48.382376 25715 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:48.382402 25050 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:48.382974 26739 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:48.383348 26785 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:48.386452 26606 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:48.387305 25050 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:48.392040 24643 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:48.398193 24786 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
W20240502 08:19:48.409502 24721 consensus_peers.cc:485] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 -> Peer 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407): Couldn't send request to peer 40fa9dc1519b476d813eb3f71e25d73c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d915caab5c694338a1a2a8ec599e1848. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:48.409543 25050 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:48.410822 25715 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:48.411491 26739 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:48.411804 26785 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:48.414710 25715 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:48.415261 26606 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:48.419230 24630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d915caab5c694338a1a2a8ec599e1848 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:48.420656 25050 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:48.421727 24643 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:48.425735 24917 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:48.427862 25715 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:48.428908 25050 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:48.429445 25845 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:48.429572 26790 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:48.430289 26790 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:48.433205 26790 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:48.435554 25845 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:48.437021 25050 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:48.438234 24643 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:48.444659 24917 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
W20240502 08:19:48.447454 24852 consensus_peers.cc:485] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf7e142110e04e9db0fdc2c5138117ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:48.447836 25845 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:48.448500 26790 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:48.451225 25050 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:48.451759 26790 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:48.452553 26729 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:48.456329 24631 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cf7e142110e04e9db0fdc2c5138117ae with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:48.457751 24643 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:48.457578 25845 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:48.464118 25050 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:48.660324 25050 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:19:48.661072 26704 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:48.686722 24786 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:19:48.687456 26704 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:48.694696 25845 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:19:48.696170 26704 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:48.940446 26876 ts_tablet_manager.cc:902] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329)
I20240502 08:19:48.942158 26876 tablet_copy_client.cc:308] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.66:35329
I20240502 08:19:48.942932 24939 tablet_copy_service.cc:140] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Received BeginTabletCopySession request for tablet cf7e142110e04e9db0fdc2c5138117ae from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:51327)
I20240502 08:19:48.943116 24939 tablet_copy_service.cc:161] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Beginning new tablet copy session on tablet cf7e142110e04e9db0fdc2c5138117ae from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:51327: session id = c809bcf4be094d1b9a931b8431fdc60c-cf7e142110e04e9db0fdc2c5138117ae
I20240502 08:19:48.945916 24939 tablet_copy_source_session.cc:205] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:48.947613 26876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf7e142110e04e9db0fdc2c5138117ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:48.952836 26876 tablet_copy_client.cc:783] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:48.953136 26876 tablet_copy_client.cc:647] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:48.955344 26876 tablet_copy_client.cc:523] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:48.958079 26876 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:48.970757 26891 ts_tablet_manager.cc:902] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: Initiating tablet copy from peer 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:48.982117 26891 tablet_copy_client.cc:308] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:34967
I20240502 08:19:48.983070 24807 tablet_copy_service.cc:140] P 3ccdae65238f49c7aa130138a4dec610: Received BeginTabletCopySession request for tablet d915caab5c694338a1a2a8ec599e1848 from peer 40fa9dc1519b476d813eb3f71e25d73c ({username='slave'} at 127.6.151.68:39129)
I20240502 08:19:48.983274 24807 tablet_copy_service.cc:161] P 3ccdae65238f49c7aa130138a4dec610: Beginning new tablet copy session on tablet d915caab5c694338a1a2a8ec599e1848 from peer 40fa9dc1519b476d813eb3f71e25d73c at {username='slave'} at 127.6.151.68:39129: session id = 40fa9dc1519b476d813eb3f71e25d73c-d915caab5c694338a1a2a8ec599e1848
I20240502 08:19:48.986333 24807 tablet_copy_source_session.cc:205] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:48.988162 26891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d915caab5c694338a1a2a8ec599e1848. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:48.994822 26891 tablet_copy_client.cc:783] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:48.995160 26891 tablet_copy_client.cc:647] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:48.997125 26891 tablet_copy_client.cc:523] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:49.000720 26891 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap starting.
I20240502 08:19:49.006821 26876 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:49.007313 26876 tablet_bootstrap.cc:492] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:49.007647 26876 ts_tablet_manager.cc:1367] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.050s	user 0.041s	sys 0.008s
I20240502 08:19:49.008536 26876 raft_consensus.cc:348] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:49.008899 26876 raft_consensus.cc:729] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:49.009159 26876 consensus_queue.cc:260] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:49.010519 26876 ts_tablet_manager.cc:1398] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.003s	user 0.002s	sys 0.000s
I20240502 08:19:49.011449 24939 tablet_copy_service.cc:342] P e2cb3e9aae7d4ef7b06b5d1e6501add5: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-cf7e142110e04e9db0fdc2c5138117ae received from {username='slave'} at 127.6.151.69:51327
I20240502 08:19:49.011596 24939 tablet_copy_service.cc:433] P e2cb3e9aae7d4ef7b06b5d1e6501add5: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-cf7e142110e04e9db0fdc2c5138117ae on tablet cf7e142110e04e9db0fdc2c5138117ae with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:49.039216 26891 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:49.039654 26891 tablet_bootstrap.cc:492] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap complete.
I20240502 08:19:49.039923 26891 ts_tablet_manager.cc:1367] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent bootstrapping tablet: real 0.039s	user 0.031s	sys 0.009s
I20240502 08:19:49.040571 26891 raft_consensus.cc:348] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:49.040807 26891 raft_consensus.cc:729] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Initialized, Role: LEARNER
I20240502 08:19:49.041013 26891 consensus_queue.cc:260] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:49.041514 26891 ts_tablet_manager.cc:1398] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:49.042531 24807 tablet_copy_service.cc:342] P 3ccdae65238f49c7aa130138a4dec610: Request end of tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-d915caab5c694338a1a2a8ec599e1848 received from {username='slave'} at 127.6.151.68:39129
I20240502 08:19:49.042685 24807 tablet_copy_service.cc:433] P 3ccdae65238f49c7aa130138a4dec610: ending tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-d915caab5c694338a1a2a8ec599e1848 on tablet d915caab5c694338a1a2a8ec599e1848 with peer 40fa9dc1519b476d813eb3f71e25d73c
I20240502 08:19:49.228837 25715 consensus_queue.cc:237] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:49.231316 25846 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:19:49.232031 26918 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:19:49.233798 24786 raft_consensus.cc:1261] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:19:49.234555 26918 consensus_queue.cc:1035] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:19:49.235303 26918 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:49.238307 25846 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:49.240144 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 17d1d1592ecc4d52addd77be44075a1b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:49.240734 24786 raft_consensus.cc:2932] T 17d1d1592ecc4d52addd77be44075a1b P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:49.241041 24643 catalog_manager.cc:5484] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 2 leader_uuid: "c809bcf4be094d1b9a931b8431fdc60c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:49.246390 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet 17d1d1592ecc4d52addd77be44075a1b with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:49.247882 26923 tablet_replica.cc:330] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:49.248121 26923 raft_consensus.cc:2229] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:49.248382 26923 raft_consensus.cc:2258] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:49.249311 26923 ts_tablet_manager.cc:1875] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:49.253764 26923 ts_tablet_manager.cc:1888] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:19:49.253899 26923 log.cc:1196] T 17d1d1592ecc4d52addd77be44075a1b P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/17d1d1592ecc4d52addd77be44075a1b
I20240502 08:19:49.254801 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet 17d1d1592ecc4d52addd77be44075a1b (table rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]) successfully deleted
I20240502 08:19:49.475858 25846 raft_consensus.cc:1203] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:19:49.563303 25715 raft_consensus.cc:1203] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:19:49.664201 25050 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:49.666478 25846 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:49.666606 24917 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:49.667112 26721 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:49.667464 26951 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:49.670172 26721 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:49.670667 24917 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:49.675129 24639 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:49.678864 25846 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:49.682966 25050 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:49.685586 25846 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:49.685781 24917 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:19:49.686434 26955 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:49.687726 26951 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:19:49.688978 24983 consensus_peers.cc:485] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11d0289cfeaa4c30a22c499c92b5b653. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:49.689306 26951 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:49.690091 25846 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:49.691607 24917 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:49.693369 24631 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 11d0289cfeaa4c30a22c499c92b5b653 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:49.695676 24639 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 7 to 8, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:50.021332 26986 raft_consensus.cc:1050] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610: attempting to promote NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c to VOTER
I20240502 08:19:50.022080 26986 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:50.024575 25846 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:50.024796 25715 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:50.025246 25050 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:50.025270 26988 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:50.025895 26941 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:50.026222 26986 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:50.029295 26986 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:50.030076 25846 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:50.033138 25049 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:50.034813 24639 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:50.041676 25715 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:50.044819 24786 consensus_queue.cc:237] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:50.046861 25846 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:50.046916 25715 raft_consensus.cc:1261] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:50.047456 26989 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:50.047981 26986 consensus_queue.cc:1035] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:50.050303 26988 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:50.050966 25715 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:50.051035 25846 raft_consensus.cc:2932] T d915caab5c694338a1a2a8ec599e1848 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:50.054822 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d915caab5c694338a1a2a8ec599e1848 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:50.056772 24639 catalog_manager.cc:5484] T d915caab5c694338a1a2a8ec599e1848 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:50.061705 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet d915caab5c694338a1a2a8ec599e1848 with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:50.067169 26993 tablet_replica.cc:330] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:50.067389 26993 raft_consensus.cc:2229] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:50.067782 26993 raft_consensus.cc:2258] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:50.068691 26993 ts_tablet_manager.cc:1875] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:50.068905 26943 raft_consensus.cc:1050] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:50.069573 26943 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:50.072216 25846 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:50.072290 25715 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:50.072978 26994 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:50.073554 26994 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:50.074285 26993 ts_tablet_manager.cc:1888] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:19:50.074421 26993 log.cc:1196] T d915caab5c694338a1a2a8ec599e1848 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/d915caab5c694338a1a2a8ec599e1848
I20240502 08:19:50.076247 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet d915caab5c694338a1a2a8ec599e1848 (table rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]) successfully deleted
I20240502 08:19:50.083740 25049 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:50.083807 26994 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.084362 26996 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:50.084618 25846 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.086992 25049 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.088348 25715 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.089486 24643 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:50.094866 24917 consensus_queue.cc:237] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:50.097343 25715 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:50.097903 26994 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:50.097954 25846 raft_consensus.cc:1261] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer e2cb3e9aae7d4ef7b06b5d1e6501add5: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:19:50.098487 26943 consensus_queue.cc:1035] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:50.100065 26996 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.100589 25715 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.104830 24631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf7e142110e04e9db0fdc2c5138117ae with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:50.105439 24640 catalog_manager.cc:5484] T cf7e142110e04e9db0fdc2c5138117ae P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.109071 25846 raft_consensus.cc:2932] T cf7e142110e04e9db0fdc2c5138117ae P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:50.109982 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet cf7e142110e04e9db0fdc2c5138117ae with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:50.110397 26993 tablet_replica.cc:330] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:50.110589 26993 raft_consensus.cc:2229] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:50.110872 26993 raft_consensus.cc:2258] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:50.111897 26993 ts_tablet_manager.cc:1875] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:50.117540 26993 ts_tablet_manager.cc:1888] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:19:50.117668 26993 log.cc:1196] T cf7e142110e04e9db0fdc2c5138117ae P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/cf7e142110e04e9db0fdc2c5138117ae
I20240502 08:19:50.118364 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet cf7e142110e04e9db0fdc2c5138117ae (table rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]) successfully deleted
I20240502 08:19:50.128785 27000 ts_tablet_manager.cc:902] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:50.130021 27000 tablet_copy_client.cc:308] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:45829
I20240502 08:19:50.130848 25070 tablet_copy_service.cc:140] P b4fd6df034eb4418ab29ad24a7a8a9fd: Received BeginTabletCopySession request for tablet 11d0289cfeaa4c30a22c499c92b5b653 from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:56025)
I20240502 08:19:50.131045 25070 tablet_copy_service.cc:161] P b4fd6df034eb4418ab29ad24a7a8a9fd: Beginning new tablet copy session on tablet 11d0289cfeaa4c30a22c499c92b5b653 from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:56025: session id = c809bcf4be094d1b9a931b8431fdc60c-11d0289cfeaa4c30a22c499c92b5b653
I20240502 08:19:50.133108 25070 tablet_copy_source_session.cc:205] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:50.134369 27000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11d0289cfeaa4c30a22c499c92b5b653. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:50.139024 27000 tablet_copy_client.cc:783] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:50.139282 27000 tablet_copy_client.cc:647] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:50.140872 27000 tablet_copy_client.cc:523] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:50.143400 27000 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:50.182610 27000 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:50.183085 27000 tablet_bootstrap.cc:492] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:50.183357 27000 ts_tablet_manager.cc:1367] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.040s	user 0.034s	sys 0.005s
I20240502 08:19:50.184093 27000 raft_consensus.cc:348] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:50.184326 27000 raft_consensus.cc:729] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:50.184545 27000 consensus_queue.cc:260] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:50.185070 27000 ts_tablet_manager.cc:1398] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.002s	user 0.000s	sys 0.000s
I20240502 08:19:50.186817 25070 tablet_copy_service.cc:342] P b4fd6df034eb4418ab29ad24a7a8a9fd: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-11d0289cfeaa4c30a22c499c92b5b653 received from {username='slave'} at 127.6.151.69:56025
I20240502 08:19:50.186957 25070 tablet_copy_service.cc:433] P b4fd6df034eb4418ab29ad24a7a8a9fd: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-11d0289cfeaa4c30a22c499c92b5b653 on tablet 11d0289cfeaa4c30a22c499c92b5b653 with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:50.391649 25049 consensus_queue.cc:237] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } }
I20240502 08:19:50.394264 24786 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:50.394280 24917 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:50.394953 27027 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:50.395344 26955 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:50.398486 27027 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:50.399122 24786 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:50.403347 24917 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:50.408143 24639 catalog_manager.cc:5484] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240502 08:19:50.413615 25049 consensus_queue.cc:237] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
W20240502 08:19:50.417039 24984 consensus_peers.cc:485] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd -> Peer 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407): Couldn't send request to peer 40fa9dc1519b476d813eb3f71e25d73c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 037baac9651a44cfba46345bba7904cc. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:50.417258 24917 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:50.417815 24786 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:50.418478 26955 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:50.419143 26955 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:50.421602 26955 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:50.422277 24787 consensus_queue.cc:237] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } }
I20240502 08:19:50.422214 24786 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:50.424844 24917 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:50.425191 25049 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:50.425473 26986 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:50.425784 26989 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:50.425830 24917 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:50.428745 24640 catalog_manager.cc:5484] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 4 to 5, NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) added. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } } }
I20240502 08:19:50.430249 26989 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:50.432459 24631 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 037baac9651a44cfba46345bba7904cc with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:50.440858 24918 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:50.441670 25049 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:50.444586 24642 catalog_manager.cc:5484] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } }
I20240502 08:19:50.450976 24786 consensus_queue.cc:237] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:50.454342 24918 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:19:50.454430 24720 consensus_peers.cc:485] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75df7ed7df824d68a02d8b8ebb92db13. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:50.454918 25049 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:50.455029 26941 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:50.455497 26989 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:50.458047 26986 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:50.458577 24918 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:50.458966 25049 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:50.463065 24630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 75df7ed7df824d68a02d8b8ebb92db13 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:50.464079 24639 catalog_manager.cc:5484] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:19:50.646008 25715 raft_consensus.cc:1203] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8]   Dedup: 1.8->[]
I20240502 08:19:50.955497 27085 ts_tablet_manager.cc:902] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: Initiating tablet copy from peer b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:50.956745 27085 tablet_copy_client.cc:308] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.67:45829
I20240502 08:19:50.957590 25070 tablet_copy_service.cc:140] P b4fd6df034eb4418ab29ad24a7a8a9fd: Received BeginTabletCopySession request for tablet 037baac9651a44cfba46345bba7904cc from peer 40fa9dc1519b476d813eb3f71e25d73c ({username='slave'} at 127.6.151.68:57367)
I20240502 08:19:50.957743 25070 tablet_copy_service.cc:161] P b4fd6df034eb4418ab29ad24a7a8a9fd: Beginning new tablet copy session on tablet 037baac9651a44cfba46345bba7904cc from peer 40fa9dc1519b476d813eb3f71e25d73c at {username='slave'} at 127.6.151.68:57367: session id = 40fa9dc1519b476d813eb3f71e25d73c-037baac9651a44cfba46345bba7904cc
I20240502 08:19:50.959954 25070 tablet_copy_source_session.cc:205] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:50.961230 27085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 037baac9651a44cfba46345bba7904cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:50.965898 27085 tablet_copy_client.cc:783] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:50.966194 27085 tablet_copy_client.cc:647] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:50.968073 27085 tablet_copy_client.cc:523] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:50.969086 27090 ts_tablet_manager.cc:902] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:50.970569 27090 tablet_copy_client.cc:308] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:34967
I20240502 08:19:50.971468 24807 tablet_copy_service.cc:140] P 3ccdae65238f49c7aa130138a4dec610: Received BeginTabletCopySession request for tablet 75df7ed7df824d68a02d8b8ebb92db13 from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:55325)
I20240502 08:19:50.971700 27085 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap starting.
I20240502 08:19:50.971681 24807 tablet_copy_service.cc:161] P 3ccdae65238f49c7aa130138a4dec610: Beginning new tablet copy session on tablet 75df7ed7df824d68a02d8b8ebb92db13 from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:55325: session id = c809bcf4be094d1b9a931b8431fdc60c-75df7ed7df824d68a02d8b8ebb92db13
I20240502 08:19:50.974694 24807 tablet_copy_source_session.cc:205] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:50.976246 27090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75df7ed7df824d68a02d8b8ebb92db13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:50.981391 27090 tablet_copy_client.cc:783] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:50.981675 27090 tablet_copy_client.cc:647] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:50.983865 27090 tablet_copy_client.cc:523] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:50.986541 27090 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:51.008191 27085 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:51.008647 27085 tablet_bootstrap.cc:492] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: Bootstrap complete.
I20240502 08:19:51.008920 27085 ts_tablet_manager.cc:1367] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: Time spent bootstrapping tablet: real 0.037s	user 0.036s	sys 0.000s
I20240502 08:19:51.009613 27085 raft_consensus.cc:348] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:51.009836 27085 raft_consensus.cc:729] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Initialized, Role: LEARNER
I20240502 08:19:51.010035 27085 consensus_queue.cc:260] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: NON_VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: true } }
I20240502 08:19:51.010850 27085 ts_tablet_manager.cc:1398] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c: Time spent starting tablet: real 0.002s	user 0.004s	sys 0.000s
I20240502 08:19:51.011930 25070 tablet_copy_service.cc:342] P b4fd6df034eb4418ab29ad24a7a8a9fd: Request end of tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-037baac9651a44cfba46345bba7904cc received from {username='slave'} at 127.6.151.68:57367
I20240502 08:19:51.012106 25070 tablet_copy_service.cc:433] P b4fd6df034eb4418ab29ad24a7a8a9fd: ending tablet copy session 40fa9dc1519b476d813eb3f71e25d73c-037baac9651a44cfba46345bba7904cc on tablet 037baac9651a44cfba46345bba7904cc with peer 40fa9dc1519b476d813eb3f71e25d73c
I20240502 08:19:51.019845 27090 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:51.020263 27090 tablet_bootstrap.cc:492] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:51.020529 27090 ts_tablet_manager.cc:1367] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.034s	user 0.026s	sys 0.007s
I20240502 08:19:51.021193 27090 raft_consensus.cc:348] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:51.021425 27090 raft_consensus.cc:729] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:51.021616 27090 consensus_queue.cc:260] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:51.022133 27090 ts_tablet_manager.cc:1398] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.001s	user 0.000s	sys 0.000s
I20240502 08:19:51.023023 24807 tablet_copy_service.cc:342] P 3ccdae65238f49c7aa130138a4dec610: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-75df7ed7df824d68a02d8b8ebb92db13 received from {username='slave'} at 127.6.151.69:55325
I20240502 08:19:51.023197 24807 tablet_copy_service.cc:433] P 3ccdae65238f49c7aa130138a4dec610: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-75df7ed7df824d68a02d8b8ebb92db13 on tablet 75df7ed7df824d68a02d8b8ebb92db13 with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:51.150631 27039 raft_consensus.cc:1050] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:51.151630 27039 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:51.154505 25846 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:51.155606 27106 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:51.157102 25715 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:51.157228 24918 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:19:51.157799 27106 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:51.159092 27106 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:19:51.162107 27039 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:51.162832 24918 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:51.163826 25715 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:51.168691 25846 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:51.169281 24640 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 8 to 9, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:51.202426 25050 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653"
dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40646
I20240502 08:19:51.202682 25050 raft_consensus.cc:595] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:51.359036 25846 raft_consensus.cc:1203] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 08:19:51.445461 25050 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653"
dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40666
I20240502 08:19:51.445678 25050 raft_consensus.cc:595] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:51.496435 25715 raft_consensus.cc:1203] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 08:19:51.633394 27106 raft_consensus.cc:979] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: : Instructing follower c809bcf4be094d1b9a931b8431fdc60c to start an election
I20240502 08:19:51.633594 27106 raft_consensus.cc:1067] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Signalling peer c809bcf4be094d1b9a931b8431fdc60c to start an election
I20240502 08:19:51.634312 25715 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653"
dest_uuid: "c809bcf4be094d1b9a931b8431fdc60c"
 from {username='slave'} at 127.6.151.67:58319
I20240502 08:19:51.634512 25715 raft_consensus.cc:482] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:51.634676 25715 raft_consensus.cc:3037] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.636744 25715 raft_consensus.cc:504] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:51.637578 25715 leader_election.cc:288] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [CANDIDATE]: Term 2 election: Requested vote from peers e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829), 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68:43407)
I20240502 08:19:51.638057 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653" candidate_uuid: "c809bcf4be094d1b9a931b8431fdc60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:51.638192 25846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653" candidate_uuid: "c809bcf4be094d1b9a931b8431fdc60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "40fa9dc1519b476d813eb3f71e25d73c"
I20240502 08:19:51.638296 24918 raft_consensus.cc:3037] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.638247 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11d0289cfeaa4c30a22c499c92b5b653" candidate_uuid: "c809bcf4be094d1b9a931b8431fdc60c" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:51.638429 25846 raft_consensus.cc:3037] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.638475 25050 raft_consensus.cc:3032] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:51.638576 25050 raft_consensus.cc:729] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Running, Role: LEADER
I20240502 08:19:51.638893 25050 consensus_queue.cc:260] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:51.639339 25050 raft_consensus.cc:3037] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:51.640254 24918 raft_consensus.cc:2454] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c809bcf4be094d1b9a931b8431fdc60c in term 2.
I20240502 08:19:51.641244 25846 raft_consensus.cc:2454] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c809bcf4be094d1b9a931b8431fdc60c in term 2.
I20240502 08:19:51.641762 25649 leader_election.cc:302] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 40fa9dc1519b476d813eb3f71e25d73c, c809bcf4be094d1b9a931b8431fdc60c, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:51.641994 25050 raft_consensus.cc:2454] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c809bcf4be094d1b9a931b8431fdc60c in term 2.
I20240502 08:19:51.642138 26861 raft_consensus.cc:2781] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:51.642540 26861 raft_consensus.cc:686] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 2 LEADER]: Becoming Leader. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Running, Role: LEADER
I20240502 08:19:51.642901 26861 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:51.645538 24643 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: term changed from 1 to 2, leader changed from b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) to c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69). New cstate: current_term: 2 leader_uuid: "c809bcf4be094d1b9a931b8431fdc60c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:51.834702 27039 raft_consensus.cc:1050] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: attempting to promote NON_VOTER 40fa9dc1519b476d813eb3f71e25d73c to VOTER
I20240502 08:19:51.835444 27039 consensus_queue.cc:237] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:51.838317 24786 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:51.838317 25846 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 1 LEARNER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:51.838526 24918 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer b4fd6df034eb4418ab29ad24a7a8a9fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:51.838956 27106 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:51.839273 27039 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:51.839576 27169 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:51.856375 27169 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:51.857184 24786 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:51.857206 24918 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:51.860569 25846 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:51.866127 24643 catalog_manager.cc:5484] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 5 to 6, 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:51.904707 25050 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "037baac9651a44cfba46345bba7904cc"
dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40686
I20240502 08:19:51.904922 25050 raft_consensus.cc:595] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:52.001554 27150 raft_consensus.cc:1050] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:52.002323 27150 consensus_queue.cc:237] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:52.004977 25715 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:52.005012 25050 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:52.005581 27195 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:52.005793 24918 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:52.005950 27150 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:52.006448 27149 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:52.010587 27150 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.011447 25050 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.015430 25715 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.016636 24918 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.017202 24640 catalog_manager.cc:5484] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.024246 24786 consensus_queue.cc:237] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:52.026057 25715 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:52.026243 24918 raft_consensus.cc:1261] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:52.026927 27150 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:52.027263 27149 consensus_queue.cc:1035] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:52.029615 27195 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.030254 25715 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.030336 24918 raft_consensus.cc:2932] T 75df7ed7df824d68a02d8b8ebb92db13 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.033735 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 75df7ed7df824d68a02d8b8ebb92db13 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:52.035018 24639 catalog_manager.cc:5484] T 75df7ed7df824d68a02d8b8ebb92db13 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.038601 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet 75df7ed7df824d68a02d8b8ebb92db13 with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:52.039551 27199 tablet_replica.cc:330] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:52.039741 27199 raft_consensus.cc:2229] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:52.040108 27199 raft_consensus.cc:2258] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:52.041090 27199 ts_tablet_manager.cc:1875] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:52.045675 27199 ts_tablet_manager.cc:1888] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:19:52.045805 27199 log.cc:1196] T 75df7ed7df824d68a02d8b8ebb92db13 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/75df7ed7df824d68a02d8b8ebb92db13
I20240502 08:19:52.046707 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet 75df7ed7df824d68a02d8b8ebb92db13 (table rebalance_test_table_1 [id=ccb8046d84dd4df2ad1b1dd5cc11b883]) successfully deleted
I20240502 08:19:52.060483 24918 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:19:52.061110 26861 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:52.143033 25050 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "037baac9651a44cfba46345bba7904cc"
dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:40694
I20240502 08:19:52.143262 25050 raft_consensus.cc:595] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Received request to transfer leadership
I20240502 08:19:52.170200 25846 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:19:52.170950 26861 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:52.176090 25050 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:19:52.177225 27223 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:19:52.180435 25715 consensus_queue.cc:237] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:52.182876 24918 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:19:52.183419 27223 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:19:52.184309 25845 raft_consensus.cc:1261] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Refusing update from remote peer c809bcf4be094d1b9a931b8431fdc60c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:19:52.185096 27223 consensus_queue.cc:1035] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [LEADER]: Connected to new peer: Peer: permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:19:52.186760 27220 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.187400 24918 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.188757 25845 raft_consensus.cc:2932] T 11d0289cfeaa4c30a22c499c92b5b653 P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:52.190495 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11d0289cfeaa4c30a22c499c92b5b653 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:52.192572 24643 catalog_manager.cc:5484] T 11d0289cfeaa4c30a22c499c92b5b653 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 9 to 11, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 2 leader_uuid: "c809bcf4be094d1b9a931b8431fdc60c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:52.196519 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet 11d0289cfeaa4c30a22c499c92b5b653 with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:52.196930 27199 tablet_replica.cc:330] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:52.197129 27199 raft_consensus.cc:2229] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:52.197307 27199 raft_consensus.cc:2258] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:52.198163 27199 ts_tablet_manager.cc:1875] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:52.202689 27199 ts_tablet_manager.cc:1888] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:19:52.202811 27199 log.cc:1196] T 11d0289cfeaa4c30a22c499c92b5b653 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/11d0289cfeaa4c30a22c499c92b5b653
I20240502 08:19:52.203465 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet 11d0289cfeaa4c30a22c499c92b5b653 (table rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]) successfully deleted
I20240502 08:19:52.254698 27098 raft_consensus.cc:979] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: : Instructing follower 40fa9dc1519b476d813eb3f71e25d73c to start an election
I20240502 08:19:52.254884 27098 raft_consensus.cc:1067] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Signalling peer 40fa9dc1519b476d813eb3f71e25d73c to start an election
I20240502 08:19:52.255846 25845 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "037baac9651a44cfba46345bba7904cc"
dest_uuid: "40fa9dc1519b476d813eb3f71e25d73c"
 from {username='slave'} at 127.6.151.67:56081
I20240502 08:19:52.256155 25845 raft_consensus.cc:482] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:19:52.256296 25845 raft_consensus.cc:3037] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:52.258365 25845 raft_consensus.cc:504] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:52.260129 25845 leader_election.cc:288] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [CANDIDATE]: Term 2 election: Requested vote from peers 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967), e2cb3e9aae7d4ef7b06b5d1e6501add5 (127.6.151.66:35329), b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829)
I20240502 08:19:52.260619 24786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037baac9651a44cfba46345bba7904cc" candidate_uuid: "40fa9dc1519b476d813eb3f71e25d73c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "3ccdae65238f49c7aa130138a4dec610"
I20240502 08:19:52.260665 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037baac9651a44cfba46345bba7904cc" candidate_uuid: "40fa9dc1519b476d813eb3f71e25d73c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5"
I20240502 08:19:52.260708 25050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "037baac9651a44cfba46345bba7904cc" candidate_uuid: "40fa9dc1519b476d813eb3f71e25d73c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd"
I20240502 08:19:52.260885 24918 raft_consensus.cc:3037] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:52.260885 24786 raft_consensus.cc:3037] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:52.260991 25050 raft_consensus.cc:3032] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:19:52.261101 25050 raft_consensus.cc:729] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4fd6df034eb4418ab29ad24a7a8a9fd, State: Running, Role: LEADER
I20240502 08:19:52.261394 25050 consensus_queue.cc:260] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:52.261770 25050 raft_consensus.cc:3037] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:19:52.263819 25050 raft_consensus.cc:2454] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40fa9dc1519b476d813eb3f71e25d73c in term 2.
I20240502 08:19:52.263926 24918 raft_consensus.cc:2454] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40fa9dc1519b476d813eb3f71e25d73c in term 2.
I20240502 08:19:52.263919 24786 raft_consensus.cc:2454] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40fa9dc1519b476d813eb3f71e25d73c in term 2.
I20240502 08:19:52.264459 25779 leader_election.cc:302] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 4 yes votes; 0 no votes. yes voters: 3ccdae65238f49c7aa130138a4dec610, 40fa9dc1519b476d813eb3f71e25d73c, b4fd6df034eb4418ab29ad24a7a8a9fd, e2cb3e9aae7d4ef7b06b5d1e6501add5; no voters: 
I20240502 08:19:52.264876 27173 raft_consensus.cc:2781] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:19:52.265290 27173 raft_consensus.cc:686] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 2 LEADER]: Becoming Leader. State: Replica: 40fa9dc1519b476d813eb3f71e25d73c, State: Running, Role: LEADER
I20240502 08:19:52.265759 27173 consensus_queue.cc:237] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:52.269281 24642 catalog_manager.cc:5484] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c reported cstate change: term changed from 1 to 2, leader changed from b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) to 40fa9dc1519b476d813eb3f71e25d73c (127.6.151.68). New cstate: current_term: 2 leader_uuid: "40fa9dc1519b476d813eb3f71e25d73c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:52.366278 24786 consensus_queue.cc:237] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } }
I20240502 08:19:52.368481 24918 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:52.368579 25050 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:19:52.369210 27195 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:52.369536 27150 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:19:52.372382 27150 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:52.373028 25050 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:52.373955 24918 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:52.378775 24642 catalog_manager.cc:5484] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } }
I20240502 08:19:52.384935 24786 consensus_queue.cc:237] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
W20240502 08:19:52.388105 24720 consensus_peers.cc:485] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 -> Peer c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69:36255): Couldn't send request to peer c809bcf4be094d1b9a931b8431fdc60c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 820325c11c654d54886546151c823403. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:52.388212 25050 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:52.388762 27150 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:52.389318 24918 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:19:52.390358 27150 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:19:52.394284 24917 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:52.395002 27149 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:52.398895 24630 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 820325c11c654d54886546151c823403 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:19:52.400885 25050 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:52.402987 24643 catalog_manager.cc:5484] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 reported cstate change: config changed from index 4 to 5, NON_VOTER c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) added. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } } }
I20240502 08:19:52.672912 24917 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Refusing update from remote peer 40fa9dc1519b476d813eb3f71e25d73c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:19:52.673663 27173 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:52.682942 25050 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Refusing update from remote peer 40fa9dc1519b476d813eb3f71e25d73c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:19:52.683594 27173 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:52.687356 24786 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Refusing update from remote peer 40fa9dc1519b476d813eb3f71e25d73c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:19:52.688268 27173 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20240502 08:19:52.942507 27277 ts_tablet_manager.cc:902] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: Initiating tablet copy from peer 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967)
I20240502 08:19:52.943913 27277 tablet_copy_client.cc:308] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.151.65:34967
I20240502 08:19:52.944666 24807 tablet_copy_service.cc:140] P 3ccdae65238f49c7aa130138a4dec610: Received BeginTabletCopySession request for tablet 820325c11c654d54886546151c823403 from peer c809bcf4be094d1b9a931b8431fdc60c ({username='slave'} at 127.6.151.69:55325)
I20240502 08:19:52.944844 24807 tablet_copy_service.cc:161] P 3ccdae65238f49c7aa130138a4dec610: Beginning new tablet copy session on tablet 820325c11c654d54886546151c823403 from peer c809bcf4be094d1b9a931b8431fdc60c at {username='slave'} at 127.6.151.69:55325: session id = c809bcf4be094d1b9a931b8431fdc60c-820325c11c654d54886546151c823403
I20240502 08:19:52.946919 24807 tablet_copy_source_session.cc:205] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:19:52.948179 27277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 820325c11c654d54886546151c823403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:19:52.952673 27277 tablet_copy_client.cc:783] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 0 data blocks...
I20240502 08:19:52.952950 27277 tablet_copy_client.cc:647] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:19:52.955026 27277 tablet_copy_client.cc:523] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:19:52.957610 27277 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap starting.
I20240502 08:19:52.989284 27277 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:19:52.989789 27277 tablet_bootstrap.cc:492] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: Bootstrap complete.
I20240502 08:19:52.990031 27277 ts_tablet_manager.cc:1367] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: Time spent bootstrapping tablet: real 0.033s	user 0.031s	sys 0.000s
I20240502 08:19:52.990721 27277 raft_consensus.cc:348] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:52.990926 27277 raft_consensus.cc:729] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c809bcf4be094d1b9a931b8431fdc60c, State: Initialized, Role: LEARNER
I20240502 08:19:52.991083 27277 consensus_queue.cc:260] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: NON_VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: true } }
I20240502 08:19:52.991557 27277 ts_tablet_manager.cc:1398] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c: Time spent starting tablet: real 0.001s	user 0.004s	sys 0.000s
I20240502 08:19:52.992394 24807 tablet_copy_service.cc:342] P 3ccdae65238f49c7aa130138a4dec610: Request end of tablet copy session c809bcf4be094d1b9a931b8431fdc60c-820325c11c654d54886546151c823403 received from {username='slave'} at 127.6.151.69:55325
I20240502 08:19:52.992600 24807 tablet_copy_service.cc:433] P 3ccdae65238f49c7aa130138a4dec610: ending tablet copy session c809bcf4be094d1b9a931b8431fdc60c-820325c11c654d54886546151c823403 on tablet 820325c11c654d54886546151c823403 with peer c809bcf4be094d1b9a931b8431fdc60c
I20240502 08:19:53.283094 25845 consensus_queue.cc:237] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } }
I20240502 08:19:53.285068 24917 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Refusing update from remote peer 40fa9dc1519b476d813eb3f71e25d73c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:19:53.285126 24786 raft_consensus.cc:1261] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Refusing update from remote peer 40fa9dc1519b476d813eb3f71e25d73c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:19:53.285617 27293 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:53.285990 27294 consensus_queue.cc:1035] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:19:53.288373 27173 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P 40fa9dc1519b476d813eb3f71e25d73c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:53.289167 24786 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:53.289193 24917 raft_consensus.cc:2932] T 037baac9651a44cfba46345bba7904cc P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:53.293839 24642 catalog_manager.cc:5484] T 037baac9651a44cfba46345bba7904cc P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 6 to 8, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 2 leader_uuid: "40fa9dc1519b476d813eb3f71e25d73c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "40fa9dc1519b476d813eb3f71e25d73c" member_type: VOTER last_known_addr { host: "127.6.151.68" port: 43407 } attrs { promote: false } } }
I20240502 08:19:53.295153 24631 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 037baac9651a44cfba46345bba7904cc with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:53.305260 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet 037baac9651a44cfba46345bba7904cc with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:53.306767 27305 tablet_replica.cc:330] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:53.306980 27305 raft_consensus.cc:2229] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:53.307214 27305 raft_consensus.cc:2258] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:53.308174 27305 ts_tablet_manager.cc:1875] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:53.314136 27305 ts_tablet_manager.cc:1888] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:19:53.314254 27305 log.cc:1196] T 037baac9651a44cfba46345bba7904cc P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/037baac9651a44cfba46345bba7904cc
I20240502 08:19:53.315042 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet 037baac9651a44cfba46345bba7904cc (table rebalance_test_table_0 [id=7fb1ecae8daa4ac2b48096ac3ca57561]) successfully deleted
I20240502 08:19:53.394726 25715 raft_consensus.cc:1203] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5]   Dedup: 1.5->[]
I20240502 08:19:53.912052 27312 raft_consensus.cc:1050] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610: attempting to promote NON_VOTER c809bcf4be094d1b9a931b8431fdc60c to VOTER
I20240502 08:19:53.912734 27312 consensus_queue.cc:237] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:53.915092 25050 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:53.915261 25715 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [term 1 LEARNER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:53.915565 24917 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:19:53.915804 27312 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:53.916188 27311 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:53.916543 27325 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:19:53.919499 27325 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.920121 25715 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.924528 24917 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.924886 25050 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.925096 24639 catalog_manager.cc:5484] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c reported cstate change: config changed from index 5 to 6, c809bcf4be094d1b9a931b8431fdc60c (127.6.151.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "b4fd6df034eb4418ab29ad24a7a8a9fd" member_type: VOTER last_known_addr { host: "127.6.151.67" port: 45829 } attrs { replace: true } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.935330 24786 consensus_queue.cc:237] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } }
I20240502 08:19:53.937290 24917 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:53.937631 25715 raft_consensus.cc:1261] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Refusing update from remote peer 3ccdae65238f49c7aa130138a4dec610: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:19:53.937932 27325 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:53.938231 27311 consensus_queue.cc:1035] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:19:53.940708 27325 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.941358 24917 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P e2cb3e9aae7d4ef7b06b5d1e6501add5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.941625 25715 raft_consensus.cc:2932] T 820325c11c654d54886546151c823403 P c809bcf4be094d1b9a931b8431fdc60c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } } }
I20240502 08:19:53.944684 24630 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 820325c11c654d54886546151c823403 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:19:53.945488 24640 catalog_manager.cc:5484] T 820325c11c654d54886546151c823403 P 3ccdae65238f49c7aa130138a4dec610 reported cstate change: config changed from index 6 to 7, VOTER b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67) evicted. New cstate: current_term: 1 leader_uuid: "3ccdae65238f49c7aa130138a4dec610" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e2cb3e9aae7d4ef7b06b5d1e6501add5" member_type: VOTER last_known_addr { host: "127.6.151.66" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccdae65238f49c7aa130138a4dec610" member_type: VOTER last_known_addr { host: "127.6.151.65" port: 34967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c809bcf4be094d1b9a931b8431fdc60c" member_type: VOTER last_known_addr { host: "127.6.151.69" port: 36255 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:19:53.952674 25030 tablet_service.cc:1514] Processing DeleteTablet for tablet 820325c11c654d54886546151c823403 with delete_type TABLET_DATA_TOMBSTONED (TS b4fd6df034eb4418ab29ad24a7a8a9fd not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40562
I20240502 08:19:53.953369 27339 tablet_replica.cc:330] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: stopping tablet replica
I20240502 08:19:53.953665 27339 raft_consensus.cc:2229] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:19:53.954113 27339 raft_consensus.cc:2258] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:19:53.956634 27339 ts_tablet_manager.cc:1875] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:19:53.961510 27339 ts_tablet_manager.cc:1888] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:19:53.961635 27339 log.cc:1196] T 820325c11c654d54886546151c823403 P b4fd6df034eb4418ab29ad24a7a8a9fd: Deleting WAL directory at /tmp/dist-test-taskOUdoRy/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx2_rf3_t3_l_3_2.1714637790512854-6749-0/raft_consensus-itest-cluster/ts-2/wal/wals/820325c11c654d54886546151c823403
I20240502 08:19:53.962368 24631 catalog_manager.cc:4830] TS b4fd6df034eb4418ab29ad24a7a8a9fd (127.6.151.67:45829): tablet 820325c11c654d54886546151c823403 (table rebalance_test_table_2 [id=4f1bc3ebb1b64cd59c3cf952a64cc366]) successfully deleted
I20240502 08:19:54.253559 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:54.287165 25695 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.292112 25030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:54.295284 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.307847 25826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:54.458819 25826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:54.459251 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:54.473628 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.485944 25695 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.496621 25030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:54.710546 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:54.724167 25030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:54.730145 25695 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.731989 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.754165 25826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:54.934608 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:54.949384 25826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:54.958956 25695 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.958982 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:54.963163 25030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:55.122560 24767 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:19:55.133741 25826 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:19:55.136682 25695 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:55.143621 24898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:19:55.145046 25030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:19:55.358124  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 24705
I20240502 08:19:55.366828 24827 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:55.529577  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 24705
W20240502 08:19:55.557865 25650 connection.cc:535] server connection from 127.6.151.65:45945 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:19:55.558004 25648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.151.65:34967 (error 108)
I20240502 08:19:55.558880  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 24836
W20240502 08:19:55.560420 25648 consensus_peers.cc:485] T 17d1d1592ecc4d52addd77be44075a1b P c809bcf4be094d1b9a931b8431fdc60c -> Peer 3ccdae65238f49c7aa130138a4dec610 (127.6.151.65:34967): Couldn't send request to peer 3ccdae65238f49c7aa130138a4dec610. Status: Network error: Client connection negotiation failed: client connection to 127.6.151.65:34967: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:19:55.565037 24959 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:55.722036  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 24836
I20240502 08:19:55.749392  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 24967
I20240502 08:19:55.756104 25090 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:55.919970  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 24967
I20240502 08:19:55.950346  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 25764
I20240502 08:19:55.957607 25885 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:56.105140  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 25764
I20240502 08:19:56.129395  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 25633
I20240502 08:19:56.166718 25755 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:56.311578  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 25633
I20240502 08:19:56.337296  6749 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu pid 24615
I20240502 08:19:56.338507 24676 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 08:19:56.511003  6749 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUdoRy/build/asan/bin/kudu with pid 24615
2024-05-02T08:19:56Z chronyd exiting
[       OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx2_rf3_t3_l_3_2 (28053 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (51037 ms total)

[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (206047 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1, where GetParam() = 4-byte object <01-00 00-00>

 1 FAILED TEST